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Abstract. Let X be a symplectic or odd orthogonal Grassmannian parametriz- 
ing isotropic subspaces in a vector space equipped with a nondegenerate (skew) 
symmetric form. We prove a Giambelli formula which expresses an arbitrary 
Schubert class in H* {X, Z) as a polynomial in certain special Schubert classes. 
We study theta polynomials, a family of polynomials defined using raising op- 
erators whose algebra agrees with the Schubert calculus on X. Furthermore, 
we prove that theta polynomials are special cases of Billey-Haiman Schubert 
polynomials and use this connection to express the former as positive linear 
combinations of products of Schur Q-functions and S'-polynomials. 



0. Introduction 



Let G = G{m, N) denote the Grassmannian of m-dimensional subspaces of i 
To each integer partition A — (Ai, . . . , Am) whose Young diagram is contained in an 
TO X {N ~m) rectangle, we associate a Schubert class a\ in the cohomology ring of 
G. The special Schubert classes ai , . . . , UN-m are the Chern classes of the universal 
quotient bundle Q over G(m, N) \ they generate the cohomology ring H*(G, Z). The 
classical Giambelli formula [G] 

(1) cr\ = dct{aXi+]-i)i,] 

is an explicit expression for a\ as a polynomial in the special classes; as is customary, 
we agree here and in later formulas that ctq = 1 and = for r < 0. 

The relation between the Schubert calculus on the Grassmannian G{m,,N) and 
the algebra of Schur's ^-functions s\ (originally defined by Cauchy [C] and Jacobi 
[J]) is well known. Given an infinite list x — {xi,X2, ■ ■ ■) oi commuting independent 
variables, we define the elementary symmetric functions er{x) by the formal relation 



oo 



[](l + a;,t)-^e,(a;)r 

i=l r=0 

and set, for any partition A, s\>{x) — det{e\^+j-i{x))ij. Here A' is the par- 
tition whose Young diagram is the transpose of the diagram of A. The ring 
A = Z[ei, 62, . . .] of symmetric functions in x has a free Z-basis consisting of the 
Schur functions s\, for all partitions A. These Schur S'- functions enjoy many good 
combinatorial properties, such as nonnegativity of their coefficients, and multiply 
exactly like the Schubert classes on G(TO,iV), when m and N are sufficiently large. 
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There is a closely analogous story to the above for the Lagrangian Grassmannian 
LG(7i, 2n) which parametrizes maximal isotropic subspaces of C^", with respect to a 
symplectic form. The Schubert classes in H*(LG, Z) are indexed by strict partitions, 
i.e., partitions with distinct (non-zero) parts, whose diagrams fit in a square of side 
n. The special Schubert classes ct,. = Cr{Q) again generate the cohomology ring, 
and there is a Giambelli-type formula due to Pragacz [Praj . This latter may be 
described in two steps: For partitions A = (a, b) with only two parts, we have 

(2) (7a,b — Cra(Tb ~ 2aa+l<Jb-l + 2(Ta+20'b-2 — ■ • • 

while for A with 3 or more parts, 

(3) (TA = PfafRan(c^A,,AJ)^<J■ 
The identities ([2]) and Q in fact also go back to the work of Schur p], who 

considered a family of symmetric functions {Q\} known as Schur Q-functions. We 
define qr{x) by the equation 



and then use the same relations ([2]) and ([3|) with qr{x) in place of Ur to define Qa,b{x) 
and then Q\{x), for each strict partition A. If we let F = 'L[qi^q2, . . .] denote the 
ring of Schur Q-functions, then the {Q\} for A strict form a Z-basis for F, whose 
algebra agrees with Schubert calculus on LG(7i,2n), as — > oo. Moreover, there 
is a well developed combinatorial theory for the Q-functions, analogous to that for 
the ^-functions. 

Choose fc > and consider now the Grassmannian IG(n — k, 2n) of isotropic 
(n — A;)-dimensional subspaces of C^", equipped with a symplectic form. We call a 
partition A k-strict if no part greater than k is repeated, i.e., Xj > k ^ Xj > Aj+i. 
The Schubert classes on IG are indexed by /c-strict partitions whose diagrams fit 
in an {n — k) x {n -\- k) rectangle. Given such a A and a complete flag of subspaces 
F. : ^ Fo C Fi C ■ ■ ■ C F2n ^ C^" such that F^+i = for < i < n, we have 
a Schubert variety 



This variety has codiniension |A| = ^Xi and defines, using Poincare duality, a 
Schubert class (Ta — [X\{F,)] in H'^'^' (IG, Z). As above, we consider the special 
Schubert classes cr^ = [-'^r(-F'.)] — Cr{Q) for 1 < r < n + A;. 

In |BKT1| . we proved a Fieri rule for the products ar(J\ in H*(IG). Equipped 
with this rule and the help of a computer, we observed that (i) when Xj < k for all j, 
then (Ta is given by the determinantal formula ([T|); (ii) when A^ > k for all non-zero 
Xj, then A is strict and a\ is given by the Ffaffian formulas It is tempting to 

ask for an analogous Giambelli formula for a\ when A is a general fc-strict partition. 
Note that the formula is determined only up to an ideal of relations; whatever the 
answer, it must naturally interpolate between the Jacobi-Trudi determinant ([T|) and 
the Schur Ffaffian A similar question was also raised by Fragacz and Ratajski 
[PR] , who were using a difi^erent set of special Schubert classes. 




Xa(F.) := {S £ IG I dim(S n Fp^(A)) >J VI < j < £(A)} , 
where £{X) denotes the number of (non-zero) parts of A and 
(4) pj{X) ■.= n + k + j - Xj - =f^{i < j : A, + A^ > 2k + j - i}. 
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The answer we give depends crucially on our choice of fc-strict partitions to 
index the Schubert classes, and uses Young's raising operators [Y, p. 199]. For 
any integer sequence a = (ai, 012, ■ • ■) with finite support and i < j, we define 
Rij{a) = (ai, . . . , Q!i + 1, . . . , ttj — 1, . . .); a raising operator R is any monomial in 
these i?ij's. Set TOq — CTq. and Rnia — for any raising operator Using 
these operators, the Giambelli formulas ([T]) and ([2])-([3]) can be expressed as 

2 R - - 

(5) o-A = - -RijO'TiA and cta = ]^ -. ''^ wa, 
respectively. 

Definition 1. For a general fc-strict partition A, we define the operator 
R^ = l[{l-R,^) H {l + R^,)'' 

where the first product is over all pairs i < j and second product is over pairs i < j 
such that \i + Xj > 2k + j — i. 

Theorem 1. For any k-strict partition A, we have a\ = R^m\ in the cohomology 
ring of lG(n — fc, 2n) . 

For example, in the ring H*(1G(4, 10)) (where fc = 1) we have 
^^321 = ^— ^(l-i?i3)(l-i?23)m32i = {l-2Ri2+2Rl^-2Rl^){l-Ri:i^R2z) m^2i 

= "^321 - 2to41i + TO42 + 2to51 - TO33 = CT3Cr20'l - 2<7i<Tl + (T4CT2 + 2ct5CTi - (t| . 

Furthermore, the theorem implies that if the fc-strict partition A satisfies Ai + Xj < 
2k + j — i for all i < j, then equation ([T|) is valid, while if Ai + Xj > 2k + j — i for 
i < j < then equations ^ and ^ hold. 

Our proof of Theorem [T] proceeds by showing directly that the expression R^m\ 
satisfies the Fieri rule for isotropic Grassmannians from (BKTl) . This is sufficient 
because the Fieri rule can be used recursively to show that a general Schubert class 
may be written as a polynomial in the special Schubert classes. The argument is 
challenging because the operator R^ depends on A, in contrast to the fixed raising 
operator expressions in ([5]) . We remark that the equations corresponding to ([5]) for 
the Schur S- and Q-functions may be deduced from the formal identities 

(6) det(x^^) = \{{x, - Xj) and Ffaffian [^l^") = 

+ Xj J . Xi + Xj 

due to Vandermonde and Schur, respectively (see e.g. [Ml F3 and IILS]). 

We next use raising operators to define a family of polynomials {Oa} indexed by 
/c-strict partitions whose algebra is the same as the Schubert calculus in the stable 
cohomology ring of IG. Fix an integer A; > and consider a finite set of variables 
y = (yi, . . . , 2/fc). For any r > 0, define dr by the equation 

oo oo 



^As is customary, we slightly abuse the notation and consider that the raising operator R acts 
on the index a, and not on the monomial ma itself. 
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SO that 'dr{x ; y) = qr-i{x)ei{y). We call F^'') := i?2, ■ ■ ■] the ring of theta 
polynomials. For any finite integer sequence a, let = Y\i "^^i , and for any fc-strict 
partition A, define the theta polynomial 

When fc = 0, we have that <d\{x\y) — Qx{x) is a Schur Q-function. As a first 
application of Theorem [1] we obtain the next two results. The first implies that 
the algebra of theta polynomials agrees with the Schubert calculus on isotropic 
Grassmannians IG(n — fc, 2n) when n is sufficiently large. 

Theorem 2. The Q\, for A k-strict, form a l^-basis ofV'^^K There is a surjective 
ring homomorphism F*^*^^ H*(IG(n — fc,2n)) such that Q\ is mapped to ax, if X 
fits inside an {n — k) x (n + fc) rectangle, and to zero, otherwise. 

Theorem 3. Let X be a k-strict partition. 

(a) // A,; + Aj < 2k + j — i for all i < j , then 

^\{x;y) = ^ S'^(a;)sA'/p'(y), where S^{x) = Aet{q^j,^+j^i{x)). 

(b) // Aj + Aj >2k+j-i for all i < j < e{X), then 

©A (a;; 2/) = ^Q,,{x)ss(^x/t,y{y), 

where the sum is over all strict partitions fj, G X such that £{fi) > £{X) — 1, and 
S{X/fi) denotes a shifted skew diagram. 

In general, we prove that the theta polynomial &xix;y) is equal to the type 
C Schubert polynomial {x, y) of Billey and Haiman |BH| indexed by the cor- 
responding Grassmannian element wx of the hyperoctahedral group (Proposition 
16. 2p . Our Giambelli formula may therefore be used to further understand these 
and related polynomials. For instance, it follows that the type C Stanley symmet- 
ric function F^j^ix) of [BH1|FK1|L] is equal to R^qx{x) (CoroUary HSl)- Moreover, 
this connection implies that the coefficients of Qxix;y) are nonnegative integers. 
These integers have several combinatorial interpretations; the one we provide stems 
from the work of Kraskiewicz |Kr) and Lam [L] . 

Theorem 4. For any k-strict partition A, the polynomial Qx is a linear combina- 
tion of products of Schur Q-functions and S -polynomials: 

where the sum is over partitions /i and v such that fi is strict and u G X with 
vi < k. Moreover, the coefficients e^^ are nonnegative integers, equal to the number 
of Kraskiewicz tableaux for wxWJ^ of shape ji. 

The definition of Kraskiewicz tableaux is recalled in fJHl In |T2) . an approach to 
tableau formulas via raising operators is applied to obtain a different expression for 
Qx{x ; y), which writes it as a sum of monomials 2"'^^^ (xy)'^ over all 'fc-bitableaux' 
U of shape A. 

We have described the theory here in the symplectic case, but there are entirely 
analogous results for the odd orthogonal groups. In fact, for technical reasons, our 
proof of Theorem [1] is obtained in the setting of orthogonal type B. We also have 
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analogues of these Giambelli formulas for the quantum cohomology rings of sym- 
plectic and odd orthogonal Grassmannians; this application will appear in jBKT2j . 
In a sequel to this paper, we will discuss the Giambelli formula for even orthogonal 
Grassmannians, which is more involved. Our results have been used in |T3) to 
obtain a Giambelli formula that expresses the equivariant Schubert classes on any 
isotropic partial flag variety as polynomials in the special Schubert classes. 

This article is organized as follows. The proof of Theorem [T] occupies il]-|j4l 
Section [5] develops the theory of theta polynomials in a manner parallel to the 
theory of Schur Q-functions, and contains our proofs of Theorems [5] and [31 Finally, 
in ij6] we show that theta polynomials are equal to certain Billey-Haiman Schubert 
polynomials for the hyperoctahedral group, and prove Theorem |4l 

1. Preliminary Results 

1.1. The Schubert varieties in IG = IG(n — fc, 2n) are indexed by fc-strict partitions 
A which are contained in an (n — fc) x (n + k) rectangle; we denote the set of all 
such partitions by T'{k, n). Consider the exact sequence of vector bundles over IG 

Q ^ S ^ E ^ Q^Q, 

where E denotes the trivial bundle of rank 2n and S is the tautological subbundle 
of rank n — k. The special Schubert class Cp is equal to the Chern class Cp(Q). 

The symplectic form on E gives a pairing iS® Q — > Oig, which in turn produces 
an injection S ^ Q* . For r > k we therefore have 

C2r(Q ® Q*) = C2r{E/S © Q*) = C2r{Q* / S) = 0, 

which impHes that the relations 

(7) al+2 (-l)V,.+,a^_, =0 for r > fc 

hold in H*(IG,Z). 

1.2. A composition a = (ai, a2, . . . , otr) is a vector of integers from the set N = 
{0, 1,2,.. .}; we let \a\ — ^ q^. For A any sequence of (possibly negative) integers, 
we say that A has length ^ if A^ = for alH > ^ and ^ > is the smallest number 
with this property. All integer sequences in this paper have finite length, and we 
will identify any integer sequence of length £ with the vector consisting of its first t 
entries. In analogy with Young diagrams of partitions, we will say that a pair [i,j] 
is a hox of the integer sequence A if i > 1 and 1 < j < Ai . 

Let A° = G N X N | 1 < i < j} and define a partial order on A° by 

agreeing that {i',j') < {i,j) if i' < i and / < j. We call a finite subset D of A° 
a valid set of pairs if it is an order ideal, i.e., (i, j) £ D implies {i',j') G D for all 
(z',/)gA° with ii',j')<{i,j). 

Any valid set of pairs D defines the raising operator 

i<j i<j ■ (i,j)£D 

Given a composition a and an integer ^ > 0, we denote by m{D, a, £) the number 
of non-zero coordinates ai such that (i,^) G D. We say that a is {D,£)- compatible 
if ai G {0, 1} whenever (i, £) ^ D. 



6 



ANDERS SKOVSTED BUCH, ANDREW KRESCH, AND HARRY TAMVAKIS 



Definition 1.1. For any valid set of pairs D and any integer sequence A of length 
£ we define a cohomology class Tx = T{D, A) recursively as follows. Set Tp — ap, 
and for an arbitrary integer sequence fi — (/ii, . . . , /if-i) and r e Z, set 

(8) T^,,. = 5](-l)l"l2"(^-"'^)T^+„r,_|„| , 

a 

where the sum is over all (13, £)-compatible vectors a £ N^~^. 

The sum ([5]) is well defined because only finitely many of its summands are 
non-zero; we also have T^^r = if r < 0. Notice that definition ([8]) of T{D, A) is 
equivalent to expanding the raising operator formula 

R''mx= II {l-R^j) n il + B.jr^l[{l-Rre) + V^,, 

i<j<e i<j<£ : {i,j)eD i=l i:(i.i)GD 

after the last (i.e., the £-th) entry of A = (/i, r). Therefore T\ = R^m\. 
1.3. If 13 = then for any integers r and s we have 

and so T^^r+i = 0, while more generally Tr^s = — Ts-i,r+i- 

We claim that if D 7^ and r, s £ Z are such that r + s > 2k, then Ts,r = —Tr,s', 
in particular Tr^r — whenever r > k. Indeed, from the definition we obtain 

and hence Ts^r — ~Tr^s whenever r + s is odd. If r + s = 2m > 2k is even, we see 
that 

(9) Tr,s + Ts,r = {-lY~^ 2 {al, -2am+l(Tm-l + 2o-™+2a™-2 ) = 

using the relations (O in the cohomology ring of IG. 

The previous observations are generalized in the next two lemmas. 

Lemma 1.2. Let A = (Ai, . . . , Aj_i) and /i = (flj-^-2, ■ ■ ■ , fJ-i) be integer vectors. 
Assume that + 1) ^ D and that for each h < j, {h,j) E D if and only if 
{h,j + 1) G D. Then for any integers r and s we have 

In particular, T\^r.r+i,p, — 0. 

Proof. If = (r, t) has positive length, we set p = (A, r, s, r) and the identity follows 
by induction, because 



a 

Therefore, we may assume that // is empty. Set £ = j -\- 1. Then we have 

a a 
a, 13 

Et T\\a\ + \P\ n-m{D,al)+m.(D,l3,t-l) rp rp rp 
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where the sums are over all (_D, f )-conipatible sequences a G W^^ and {D^i — 1)- 
compatible sequences (5 £ W'"^ . The assumptions on D imply that these two sets 
of sequences coincide, and this proves the lemma. □ 

Lemma 1.3. Let A — (Ai, . . . , Aj_i) and /i ~ (/ij-|_2, ■ • ■ , A'f) ^6 integer vectors, 
assume + 1) D, and that for each h>j + l, {j,h)GD if and only if 
(j ' + 1, /i) e Z?. If r, s £ Z are such that r + s > 2k, then we have 

In particular, T\^r,r,fj. = for any r > k. 

Proof. If /i = (r, i) has positive length, we set p = (A,r, s,t) and p' — (A,s,r, r), 
and the identity follows by induction, because 

Tp.t = ^(-l)l"l2'"(^'"'^)Tp+„Tt_|„| = -^(-l)l"l2"(^'"'^)rp,+„rt_|„| = ~Tp,^t. 

a a 

Thus we may assume that n is empty. Set £ = j + 1, and note that {h,h') e D 
for all h < h' < £. If to > is the least integer such that 2m > £, we claim that 

Tn = T\ r t satisfies the relation 



2m 

(10) t, = Y,{-iYt,,^,^t,,_p,_p,^ 

1=2 

Equation ([T0|) follows from the formal identity of raising operators 

n l — Rhh' _ V^. , y 1 — Rli TT 1 — Rhh' 

l + Rhh' ^ ' 1 + Rm ^\ l + Rhh' 

l<h<h'<2m 1=2 2<h<h' <2m 

which is equivalent the classical formula 



n = Pfafiian 



l<h,h'<2r. 



due to Schur [Si Sec. IX]. The proof is completed using induction, starting from the 
base case of j = 1, which was obtained in □ 

During the above discussion the set D has remained fixed, but in subsequent 
arguments we will need to modify it. For this, we use a simple observation. 

Lemma 1.4. // {i,j) ^ D and D U {i,j) is a valid set of pairs, then 

T{D, A) = T{D U (i, j). A) + T{D U (z, j), i?.jA). 

Proof. The assertion follows immediately from the identity 

1 — Rij 1 — R-ij 

^'^^'^TTr~^TTr~^''- ° 
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2. From IG(n - k, 2n) TO OG(n - fc, 2n + 1) 

2.1. For each /c > 0, the odd orthogonal Grassmannian OG — OG{n — k,2n + 1) 
parametrizes the (n — /c)-dimensional isotropic subspaces in C^"^^, equipped with 
a nondegenerate symmetric bihnear form. Our aim is to show that if A is any Re- 
strict partition, then a\ is given by the raising operator expression of Theorem [T] 
For technical reasons, we will use an isomorphism to transfer this relation to the 
cohomology ring of OG, and work with the latter space. 

The Schubert varieties in OG are indexed by the same set of fc-strict partitions 
V{k,n) as for IG(ri — k,2n). Given a complete flag F, of subspaces of C'^''^^ such 
that Fn+i = for 1 < i < n + 1 and A € V{k, n), we define the codimension 

|A| Schubert variety 

X^{F.) = {E e OG I dim(S n Fp^(A)) >j VI < j < £(A)} , 

where 

(11) p^ (A) =n + k + l+j-Xj-#{i<j:X, + \j > 2k + j - i}. 

Let ta G H^'''*'(0G, Z) be the cohomology class dual to the cycle given by Xx{F,). 

For any A G 'P{k,n), let £fc(A) be the number of parts A^ which are strictly 
greater than k. Let Qig and QoG be the universal quotient vector bundles over 
IG(n — fc, 2n) and OG(n — fc, 2n + 1), respectively. It is known (see e.g. jBSl §3.1]) 
that the map which sends dp — Cp(QiG) to Cp(QoG) for all p extends to a ring 
isomorphism ((> : H*(IG,Q) H*(OG,Q). Moreover, we have (t>{ax) = 2^'=(^Va for 
all A G V{k,n). 

We let Cp = Cp(Qog)- The Chern classes Cp are related to the special Schubert 
classes Tp on OG by the equations 




Tp if p < fc, 
2rp if p > fc. 



Using the isomorphism 0, we can therefore describe the Giambelli formula for 
OG(n — fc, 2ri + 1) as follows. For any integer sequence a, set nia — Hi ^ai] then 
for every A G V^k^n), we have 

(12) rA = 2-^^(^)i?^mA 

in H*(OG,Z). 



2.2. For A any fc-strict partition, we say that the box [r, c] in row r and column c 
of A is k-related to the box [r' , c'] if |c — fc — 1| + ?- = |c' — fc — 1| -I- r'. If c < fc < c', 
then this is equivalent to c + c' = 2fc + 2 + r — r'. For example, in the partition 
displayed below, the grey box [r, c] is fc- related to [r',c']. The notion of fc- related 
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boxes makes sense also for boxes outside the Young diagram of A. 

k 





(r, c) 












s 

s 





Given two Young diagrams [i and v with /i C i^, the skew diagram v j [i is called a 
horizontal (resp. vertical) strip if it does not contain two boxes in the same column 
(resp. row). For any two fc-strict partitions A and /i, we write A — s> /i if /i may be 
obtained by removing a vertical strip from the first fc columns of A and adding a 
horizontal strip to the result, so that 

(1) if one of the first fc columns of /i has the same number of boxes as the same 
column of A, then the bottom box of this column is fc-related to at most one box 
of /i \ A; and 

(2) if a column of /i has fewer boxes than the same column of A, then the removed 
boxes and the bottom box of /i in this column must each be fc-related to exactly 
one box of \ A, and these boxes of /i \ A must all lie in the same row. 

Equivalently, X ^ fi means that A^ — 1 < fij < Xj-i for each j, Xj < fij when 
Xj > k, and conditions (1) and (2) are true. Let A be the set of boxes of /i \ A in 
columns fc+1 through fc+n which are not mentioned in (1) or (2), and define ^{X, fj.) 
to be the number of connected components of A. Here two boxes are connected if 
they share at least a vertex. In jBKTli Theorem 2.1] we proved that the Fieri rule 

(13) Cp-r.= 2'^^'''^^r^ 

X^fj,, \fj,\ = \X\+p 

holds, for any p G [l,n + k]. 

2.3. A comparison of ^ with suggests modifying the definition of valid sets of 
pairs from Sjljto include elements along the diagonal \ i > 0}. This convention 

will make the formalism of our proof of Theorem [T] cleaner, and is in fact crucial in 
the corresponding proof of Giambelli for even orthogonal Grassmannians. 

Set A = {(i,j)gNxN| 1 < i < j} with the same partial order as in §1.2) and 
define the notion of a valid set of pairs exactly as before. Given a fc-strict partition 
A and an integer t > £{X), we obtain a valid set of pairs Ct{X) by 

Ct(A) = e A \ Xi + Xj > 2k + j -i and j < t}. 

Furthermore, we let C(A) = C£(a)(A). Notice that C(A) includes the pairs such 
that Xi > k. It is easy to see that when fc > 0, a set D C A is a valid set of pairs if 
and only if there exists a fc-strict partition A for which C(A) — D. 

An outer corner of a valid set of pairs _D C A is a pair G A \ D such that 
D U is also a valid set of pairs. The outside rim dD of D is the set of pairs 
(i, j) e A \ £» such that i = 1 or (i - 1, j - 1) G L». 
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Lemma 2.1. Let ^ he a k-strict partition such that A — > /x. Then for any t > i{X), 
we have Ct(A) C Ct+l{^^) ^ Ct(A) U dCtiX). 

Proof. If (i, j) G Ct+i{p), then Ai_i + Aj_i > /ii + ^ij > 2k + j — i. This proves 
that Ct+i(ju) C Ct(A) U dCt{X). If there exists a pair (i, j) £ Ct{\) \ Cf+i(/i), then 
Ai + Aj > 2k + j — i > iii + ^ij, so we must have fit = Xi, fij = Aj — 1, and 
\i + Xj = 2k + 1 + j — i. Condition (2) of i j2.2l impUes that some box [h, c] of /i \ A 
is fc-related to [j, Aj], and [h,c — 1] is also in /i \ A since this box is fc-related to 
[j — l,Aj]. The equahty Xj + c — 2k + 2 + j — h imphes that {h,j) G Ct+i{n), 
and since Ct+i{p-) is a vahd set of pairs, we must have h < i. But we also obtain 
Xh<c— I — 2k + l+ j~h~Xj — Xi + i — h, contradicting the fact that A is 
fc-strict. This proves that Ct{X) C Ct+i{fJ,)- □ 

Definition 2.2. For any valid set of pairs D C A and any integer sequence A we 
define the cohomology class T{D, A) e H*(OG) by 

T{D, A) = 2-*^^' I 0(T(D n A°, A)), 

where T{D n A°, A) G H*(IG) is defined by (0). 

To prove and hence also establish Theorem [TJ it suffices to show that if A 
is a fc-strict partition, the Fieri rule 

(14) c,-r(C(A),A)^ 2^^''^^r(C(A^),M) 

X^fj., \fj.\ = \X\+p 

holds in II*(OG,Z), for all p. To see this, write ^ y X ii fi strictly dominates A, 
i.e., /X 7^ A and fii + ■ ■ ■ + fii > Ai + • • • + Ai for each i > 1. We deduce from p3)) 
and (Ull) that 

2^.(A) rx + J2 «Am = CAi • • • CA, = 2^'=^^) r(C(A), A) + ^ T{C{fi), pi), 

for some constants a^^ G Z. By induction on A, it follows that T\ = r(C(A),A), 
which is a restatement of (|12p . 

Observe that Lemmas II. 21 11.31 and ll.4l carrv over verbatim to our current setting 
where D C A. These lemmas are the main properties of the cohomology classes 
T{D, A) that we use, and as such constitute the technical core of our proof of 
Theorem [TJ But the non-trivial scheme that puts them to work together is an 
algorithm with a substitution rule; this is explained in the next section. 

3. The Substitution Rule 

3.1. Throughout the next two sections we fix p > 0, the fc-strict partition A of 
length £, and choose n sufficiently large so that we can ignore it in the sequel. Set 
C = C(A). For any d> 1 define the raising operator i?^ by 

Rd= n (1-%) n (1 

l<i<j<d i<j:(ij)eC 

We compute that 

e 

Cp ■ T{C, X) = cp- 2-'''(^^^R^ mx = 2-''^^^R^+, ■ [](1 - R^^e+iV^ mx^p 

i=l 
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Figure 1. A valid set of pairs C (white dots) and a subset of dC 
(grey dots). 

t 

= 2-^''(^)i?,\i • [](1 + + + . . . ) mA,p 

1=1 

and therefore 

(15) Cj,-T{C,\)=Y,T{C,u), 

where M — Af{X,p) is the set of all compositions ly > X such that = |A| +p and 
Vj = for j > + 1. Our strategy for proving Theorem [1] is to show that the right 
hand side of equation (|15p is equal to the right hand side of the Fieri rule ^T^ . 

3.2. The following objects will be used as book keeping tools in a delicate process 
of rewriting the right hand side of (jisp . Let m > 1 be minimal such that Am l£ 
we call m the middle row of A. Notice that m is the smallest positive integer for 
which (m, m) ^ C. 

Definition 3.1. A valid 4-tuple of level /i is a 4-tuple ij} = {D, /j,, S, h), such that 
h is an integer with 0</i<£ + l,-Disa valid set of pairs containing C, all 
pairs {i,j) in D satisfy i < m and < ^ + 1, S* is a subset of _D \ C, and fi is 
an integer sequence of length at most £ + I. The evaluation of ip is defined by 
ev(V') =r(AM) 6H*(0G,Q). 

All valid 4-tuples encountered in this paper will also satisfy that D dCDdC (see 
Lemma |4J|) . but for technical reasons we do not require this in the definition. We 
will represent the set A as the positions on or above the main diagonal of a matrix, 
and the various sets of pairs D as sets of entries in this matrix. In Figure [T] the 
white dots represent a set of pairs C, the grey dots are a subset of the outside rim 
of C, and we have m = 10. The union of the white and grey dots form the set D in 
a typical valid 4-tuple (D, /z, S, h). 

In the following we set /iq = oo whenever /i is an integer sequence. 

Definition 3.2. For any y G Z we let r{y) denote the largest integer such that 
r{y) <i + l and Xr{y)~i > 2k + r{y) - y. 

Since Aq = cx) we have r{y) > 1. Notice that for G A and j < £ we have 
(i, j) G C <^ Aj > 2k + j — i — Xi ^ j < r{i + Xi + 1). This gives the relation 

(16) C = {(^,J)G A|j<r(z + A, + 1)}. 

The function r{y) is also connected to the notion of fc-relatedness of boxes. 
Assume that some box [i,c] with c > k is /c-related to a box [j,d] in the first k 
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Figure 2. The set C near the pair x — {h,gh) G dC. 

columns of A. Then Xj >d = 2k + 2+ j — i — c, which imphes that j < r{i + c). 
Furthermore, if [j + 1, d + 1] ^ A then r(i + c) = j + 1. 

Definition 3.3. Let ft G N satisfy 1 < ft < m and let /x be an integer sequence. 

(a) We define hh — r{h + Xh + 1) and — By convention we set gi = i + 1. 

(b) Set = {[i, c] G /I \ A I c > fc and Hr(i+c) < 'ik + r{i + c) ~ i ~ c}. 

(c) Assume that ft > 2 and fih > Xh-i- If [ft, A;i_i] G R{fJ.) then set e?i(^) = Xh-i- 
Otherwise choose ChifJ-) > maxjfc, X^} minimal such that [ft,c] ^ R{p) for ehifJ-) < 
c < Xh-i- Finally, set fh{n) ^ r{h + eh{n)). 

Notice that for ft < m we have bh = min{j > m | {h,j) ^ C}. The integers bh 
and gh are illustrated in Figure [21 In the definition of R{fJ.), suppose that some box 
[i, c] d fi \ X with c > fc is fc-related to a box [j, d] in the first k columns of A, such 
that [j + 1, d + 1] ^ A. Then we have r{i + c) = j + I and d = 2fc + 2 + j - i - c. 
It follows that [i, c] G R{p) if and only if fij+i < d. In particular, if /x is a k- 
strict partition such that A ^> /i, then the set A from i i2. 21 consists of the boxes of 
fi \ X in columns fc + 1 and higher which are not in R{p-). Notice also that since 
^ + A/i + l<ft + e^(/i) < ft + A/i_i and rijj) is a monotone increasing function of 
y, we always have 

(17) bh < M^i) < gh ■ 

In particular we have fhifJ-) > m and (ft, fh{p)) ^ C; when h — m the inequality is 
true because Am-i > 2fc — ShifJ-) = 2fc + rn — ft — ehifJ-)- Furthermore, if [ft, A^-i] G 
R{^i), then /^(At) = gh- 

Example 3.4. Let fc 3, A = (9, 7, 3, 2, 1, 1), and /i = (11, 12, 7, 2, 2). Then 

C(A) = {(1, 1), (1, 2), (1, 3), (1, 4), (2, 2), (2, 3), (2, 4)}. 

Figure [3] illustrates A and fi, with the boxes in ^ \ A shaded, and the boxes in i?(/i) 
marked. Note that there is one box in A \ /x. We have e2{fJ-) = 8, /2(a*) = 5, 92 — 5, 
63(^4) = 6, fsin) = 4, and .93 = 5. 

Lemma 3.5. If 2 < h < m then we have Xh-i — Xh > gh ^ bh + 1- 

Proof. The inequality is clear if bh ~ gh, as A is fc-strict and ft < rn. If bh < gh, then 
since bh = r{h + Xh + I) and gh = r(h + A/i_i) we have Af,^ < 2k + bh — h — Xh and 
Ag^ _ 1 > 2fc + g/,. - ft - Aft- 1 , which implies that A,i_ 1 - Xh > gh - Ag^ _ 1 - 6/, + A^^ > 
.9/1 - bh- □ 

Lemma 3.6. Let fi be an integer sequence and 2 < ft < m. If ^ih ^ Xh-i and 
Xh-i + Msh <'^k + gh~h, then fh{n) = gh- 
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Figure 3. The shapes A and ^, with /i \ A shaded. 

Proof. We have A/i_i > k and [h,Xh-i] G /i \ A. Since gh = r{h + A/i_i), the 
inequahty fig^ < 2k + ^ h — X^-i shows that [h, A/i_i] G Ri^J■)■ This imphes that 
eh(M) = ^h-i and /^(/i) = r{h + A/i_i) = gh, as required. □ 

Lemma 3.7. Let 2 < h < m and let fi and fi' be integer sequences such that 
l^h > Ah-i, ^'^ > A/i_i, and fij = fi'j for max(m, h+1) < j < gh- Then [h, c] G 
«/ and only if [h, c] G R{lJ^') for all c < Xh-i- In particular, we have eh{n) = e/i(/x') 
and fhin) = 

Proof. Let [h,c] G fi \ X satisfy k < c < Xh-i, and set j — r{h + c). Since fc- 
strictness of A imphes that k + m + l<h + c<h + Xh-i, we obtain m < j < gh 
and hence /ij = /i^- provided that j > h. It follows that [/i, c] G R{^J-) if and only if 
[h,c] G as required. □ 

If we are given a fixed valid 4-tuple {D,fi, S, h) with 1 < /i < m, we will use 
the shorthand notation b = bh, g = gh, R = R{p), e = eh{p), and / = the 
values e and / will be used only when fih > A^-i. The precise value of / will play a 
crucial role in our proof that the Fieri terms in (IT4l) appear in dTSl) with the correct 
multiplicities. For example, it is part of the following definition of a condition X, 
that will be used to identify undesired valid 4-tuples. 

Definition 3.8. Let {i,j) G A be arbitrary. We define two conditions W(i,j) and 
X on a valid 4-tuple {D, ji, S, h) as follows. 

W(i,j) : Hi + > 2k + j - i . 

Condition X is true if and only if {h, h) G D and 

tih > IJ-h-i or /i/, > A,i_i or {^ih = Xh-i and {h, f) ^ S) . 

3.3. The following substitution rule will be applied itcratively to rewrite the right 
hand side of (fT5|). It may be applied to any valid 4-tuple of positive level and 
will result in either a REPLACE statement, indicating that the 4-tuple should be 
replaced by one or two new 4-tuples, or a STOF statement, indicating that the 
4-tuple should not be replaced. 

Substitution Rule 

Let [D, fj,, S, h) be a valid 4-tuple of level h > 1. Assume first that {h, h) ^ D. If 
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(i) there is an outer corner {i, h) of D with i < m such that W(i, h) holds 
then REPLACE {D, fi, S, h) with 

{DU {i,h),ii,S,h) and {D U {i,h), R^hH, S U {i, h),h). 

Otherwise, if 

(ii) D has no outer corner in column h and /x/j > Xh-i, 

then STOP. 

Assume now that {h, h) e D. If 

(iii) there is an outer corner {h,j) of D with j < £ + I such that W(/i, j) holds, 

then REPLACE (£», /i, S, h) with 

iDUih,j),ii,S,h) and (D U (/i, j), i^/yA*, U (/i, j), /i) if < /^^-i, 
(D U {h,j),Rhj^i, S U {h,j),h) if /^^ > /Xj_i. 

Otherwise, if 

(iv) W{h,g) or X holds, and D has an outer corner {i,g) with i < h, 
then REPLACE (£>, /i, 5, /i) with 

{DLl{i,g),fi,S,h) and {DU {i,g),Rigfi,S U {i,g),h). 

Otherwise, if 

(v) X holds, 

then STOP. 

If none of the above conditions hold, REPLACE (£>, fi, S, h) with (Z), ^,S,h — l). 

Definition 3.9. Let (x) be one of the conditions (i)-(v) of the Substitution Rule. 
We say that a valid 4-tuple V' meets condition (x) if -0 reaches condition (x) in the 
Substitution Rule, and condition (x) is satisfied. Whenever the Substitution Rule 
REPLACES -0 by one or two 4-tuples V'i, we refer to %}j as the parent term and the 
ij^i are its children. 

3.4. Initially, we define the set = {(C, 0, ^+1) | v e N{\p)]; thus ^v-g* 
agrees with the right hand side of (jisp . We then apply an algorithm which will 
change this set by replacing some 4-tuples with one or two new valid 4-tuples. The 
algorithm applies the Substitution Rule to each element (D, /x, S, h) of level h>l. 
If the substitution rule results in a REPLACE statement, then the set is changed 
accordingly; otherwise the substitution rule results in a STOP statement, in which 
case the 4-tuple {D, /i, 5", h) is left untouched. These substitutions are iterated until 
no further elements can be REPLACED, i.e., until the substitution rule results in 
a STOP statement when applied to any remaining 4-tuple with h>\. 

Since the set of pairs D is not allowed to grow beyond column £-1-1, the algorithm 
will terminate after a finite number of steps. Notice that if "0 = {D, fi, S, h) is any 
4-tuple produced by the algorithm, then the initial 4-tuple -00 = (C, j^, 0, £ -I- 1) that 
gave rise to tp can be recovered by the equation v = Y[(^i j)^s ^ijl^- Here Lij denotes 
the lowering operator which is the inverse of Rij . Furthermore, the sequence of 4- 
tuples leading from to ip is uniquely determined by ip because all choices made 
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along the way are recorded in the set S. In particular, no 4-tuple can be produced 
multiple times. 

Suppose that the 4-tuple ip = {D,fi, S, h) occurs in the algorithm. If ip is RE- 
PLACED by two 4-tuples ipi and -02, we deduce from Lemma [1.41 that ev(V') = 
ev(?/ji) -f ev(V'2)- Moreover, if ip meets (iii) and is REPLACED by the single 4-tuple 
^' = (DU {h,j),Rhjfi,SLI {h,j),h), then Lemmas [L2l and fOl imply that ev(V') = 
ev{ip')- Indeed, it follows from Corollarv l4.10l below that /ij-i = /^j — 1 and DL){h, j) 
has no outer corner in column j, so Lemma FO] shows that cv{D\j{h, j), /i, S, h) — 0. 

When the algorithm terminates, let 5*0 (respectively ^i) denote the collection 
of all 4-tuples {D, ji, S, h) in the final set such that h = (respectively h > 0). We 
say that a 4-tuple ip survives the algorithm if at least one of its successors lies in 
^0- The above analysis implies that 

veN i/'G*o iAG*! 

In the next section, we will prove the following two claims. 

Claim 1. For each 4-tuple ip — {D,^,S,0) in with > 0, /i is a fc-strict 

partition with X ^ fi and ev('0) = T{C{p), fj,). Furthermore, for each such partition 
/i, there are exactly 2^^'*''''^ such 4-tuples ip^ in accordance with the Fieri rule. 

Claim 2. There exists an involution i : — !> vji-^ of the form l{D, /i, S, h) = 
(D, S', h) such that ev(V') 4- ev{L{ip)) = 0, for every t/' e *i. 

We remark that the 4-tuples V' G ^0 with /i^+i < evaluate to zero trivially, by 
Definition 11.11 The two claims therefore suffice to prove the Fieri rule ([T4| . 

For each initial 4-tuple ipo = (C, i^, 0, ^-1-1) of the sum (fTSj). the algorithm produces 
a tree of 4-tuples with root node given by ipQ. If the Substitution Rule REPLACES 
a 4-tuple by one or two other 4-tuples tpi, we have a branch in the tree from ip 
to the ^pi. The leaves of the tree are exactly the 4-tuples with /i = or where the 
Substitution Rule STOFS. The fate of all the terms of the sum (fT5|) is encoded by 
the collection of all the trees with root nodes (C, v,9,£+ 1) for i/ G J\f{X,p). This 
collection will be called the substitution forest] the sum of the cohomology classes 
represented by the roots of the substitution forest is equal to the sum of classes 
given by the leaves. 

Example 3.10. We discuss an example of the substitution forest in detail. Con- 
sider the Grassmannian OG(n ~ l,2n + 1) for n > 5, and the Fieri product 

Cl • T24,l = T2,14,l + 2 T3^l4 -I- T5. 

For simplicity, we will omit the commas in our notation for compositions and 
pairs. Thus A = 211, k — p — 1, and we have C(A) = {11} and A/'(A,p) = 
{2111,2120,2210,3110}. The substitution forest is pictured in Figure H except 
that we have omitted those nodes {D, /i, S, h) which have (£*, /i, 5*, /i -I- 1) as parent 
and {D, ^^S^h — 1) as child. 

Observe that the root ({11}, 2120, 0, 4) is the only initial 4-tuple that does not 
survive the algorithm. We have *o = { ({11}, 2111, 0, 0), ({11, 12}, 3110, {12}, 0), 
({11, 12}, 3110, 0, 0), ({11, 12, 13}, 5000, {12, 13}, 0) }, which corresponds exactly to 
the terms in the Fieri product ci •T211. Furthermore, each 4-tuple in the set = { 
({11, 12, 22}, 2210, 0, 2), ({11, 12, 22}, 2210, {22}, 2), ({11}, 2120, 0, 3) } evaluates to 
zero in the cohomology ring of OG. 
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{{11}, 2210, 0, 4) {{11}, 2111, 0, 4) 




{{11, 12, 22}, 2210, {22}, 2) 
(v) 



{{11}, 2120, 0, 4) {{11}, 3110, 0, 4) 




({11, 12}, 3110, 0, 2) {{11, 12}, 4010, {12}, 2) 




({11, 12}, 3110, 0,0) {{11, 12}, 4010, {12}, 1) 

1""' 

({11, 12, 13}, 5000, {12, 13}, 1) 

i 

({11, 12, 13}, 5000, {12, 13}, 0) 



Figure 4. The substitution forest for A = 211 and k ~ p — 1. 

4. Proof of Theorem [T] 

4.1. Recall the fixed choices of p, A, £, C, and m from ii3.ll In ij4. II through H4.3\ 
we furthermore let V' = {D, fj,, S, h) denote a 4-tuple which occurs at some step in 
the algorithm, i.e., a node of the substitution forest. The symbols D, S, h will 
refer to components of the 4-tuple ij}. We will occasionally work with more than 
one valid 4-tuple. If (D', /x', 5", h') is an additional 4-tuple, then the sets and values 
that Definition 13.31 associates to this 4-tuple will be called i?', e', /', and g' . 

The algorithm has two phases. A 4-tuple ij) is in Phase 1 if [h, h) ^ D, and in 
Phase 2 if (/i, h) 6 D. The level h is always used to index an entry of the integer 
sequence fiintp; it begins at h = £+1 and decreases as the 4-tuple proceeds through 
the algorithm. In Phase 1 we have h > m, while h < m in Phase 2. Throughout 
the algorithm we have i < m < j for each € S, so ^ is obtained from the 

initial composition v by removing boxes from rows weakly below the middle row of 
A and adding them to rows weakly above the middle row. 

The set D is initially equal to C and grows when REPLACE statements are 
encountered. Lemma 14.11 below shows that all pairs added to D come from the 
outer rim dC. In Phase 1, pairs are added by rule (i) to column h, so as the level 
h decreases from £ -I- 1 to m, these pairs are added along vertical columns of dC, 
proceeding from top (row 1) to bottom (row m) and right to left. In Phase 2, the 
set D mainly grows when rule (iii) adds pairs to row h, in which case the pairs are 
added in horizontal rows of 9C, from left to right and bottom to top. In some cases 
rule (iv) will add extra pairs {i,g) to D, where i < h. Lemma [4.61 implies that if 
'ijj meets (iv), then it will not survive the algorithm, and only pairs from column g 
of dC can be added to its successors. In particular, all 4-tuples in are produced 
from the the initial 4-tuples by applications of rules (i) and (iii) . 

Our proof of Theorem [1] occupies the remainder of this section. In HA.2\ we 
prove some properties satisfied by 4-tuples that occur in the algorithm. Additional 
properties for 4-tuples in ^'q are proved in H4.3[ The proof of Claim [T] is then given 
in fj431 while Claim [2] is justified in gH 
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4.2. We prove some lemmas that reveal what can happen to the 4-tuple ip = 
(D, /i, S, h) durmg the algorithm. 

Lemma 4.1. We have D cCUdC. 

Proof. It is enough to show that if the substitution rule adds the pair to D, 
then € dC. Notice first that i < h < j. If ^ dC, then i > 1 and 

{i — 1, j — 1) ^ C. Let i/y = (D', fi' , S' , h') be the most recent predecessor of ■0 such 
that {i — l,j) ^ D' . Then ij}' meets rule (i), (iii), or (iv), which adds the pair 
(i — 1, j) to D' . Since the pair (i — 1, j — 1) ^ D' \C was added to a predecessor of 
tp' of level smaller than j, it follows that i < h < h' < j — 1, so tp' does not meet 
(i) or (iii). But ip' also does not meet (iv) because g' < j — I, a contradiction. □ 

Lemma 4.2. If j > h and ^ D, then fij < Aj_i. 

Proof. Assume that /ij > Aj_i and let tp' = {D' , fi' , S' , j) be the most recent 
predecessor of ^ of level j. Then > fij, and since tp' does not meet (ii), it follows 
that D' has an outer corner in column j. If i < j, then since — 1) e C we 
obtain /i^ + /i^ > Xj-i + Xi > 2k + (j — 1) — i, and otherwise we have i — j — m 
and ii'j > Xm-i > k. In both cases ip' satisfies W(z,j). But then ip' meets (i) and 
is not the most recent predecessor of tp of level j, a contradiction. □ 

Lemma 4.3. If 2 < h < m, ^ Xh-i, and f < g, then /ig ~ Ag_i and (h, g) ^ 5*. 

Proof. By assumption we have h < ni < f < g, and Lemma 14.21 implies that 
Mg < Ag_i < Am < k. Set x 2/c + g — /i — fig. Since {h, f) ^ C we get 
{h,g — 1) ^ C which implies \h < x. Since [h, Xh-i] ^ R we also obtain x < Xh~i- 
Assume that fig < Ag_i. Then we get Ag__i > 2k + g — h — x, which implies that 
g < r(h + x) < r{h + Xh-i) = g. The definition of x now shows that [h,x] G R, 
from which we deduce that x < e and g — r{h + a;) < r{h + e) — f, a contradiction. 
Finally assume that {h, g) G S. Since [h^g —1) ^ C we deduce that the parent of 'ip 
is Ip' = {D \ (h, g), ii' , S \ {h, g), h), where /i' = Lhgfi- But Lemma 4.2 then implies 
that /ig < /ig < Ag_i, a contradiction. □ 

We next make some observations concerning condition X and rules (iv) and (v) . 

Lemma 4.4. // condition X holds for ip, then X also holds for the children of ip>. 
In particular, ^ does not survive the algorithm, and all its successors have level h. 

Proof. Assume that -0 = {D, /i. S, h) satisfies condition X and let %p' — {D' , /i', S", h) 
be a child oi ip. If S" = then fi' — fi and X also holds for ip' , so we may assume 
that S' \ S = {{i,j)} and ^' = RijH, where i < h. We can also assume that 
A/i_i = fih = f-'h < m5i-i- Since {h,h) G D and (i,j) ^ D, we get i < h < j. 
In particular, ip meets (iv) and j — g. Let ip — {D, Jl, S, g) be the most recent 
predecessor of ip of level g, and let {a,g) be the outer corner of D in column g. 
Then a < h — 1, and W{a,g) fails for ip since it does not meet (i). Using this and 
/c-strictness of A, we obtain Xh-i + fig < Xh-i + fig < {Xa + a — h + 1) + Jig < 
Jig^ + Jig + 1 + a — h < 2k + 1 + g — h. Lemma 13.61 now implies that /' — g. We 
conclude that tj/ satisfies condition X as /i'^ — Xh-i and {h,f') — {h,g) ^ S'. □ 

Lemma 4.5. Assume that {h, h) d D and ip satisfies X or W{h,g). 

(a) If h < g and {h,g — \) ^ D, then ip meets (iii). 

(b) If Ip meets (v), then {h,g) G D. 
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Proof. Assume that h < g and {h,g — 1) ^ D, and let {h,d) be the outer corner of 
D in row h. Then d < g — 1 and {h — l,d) G C. Using Lemma l42l and the fact that 
C and D have equally many pairs in column d, we obtain /ig < Xg-i < A^j < /i^. If 
ijj satisfies condition X, then /z^ > Xh-i and it follows that ij-h + l^-d ^ ^h-i + Ki > 
2k + d—h+l. If tp satisfies W(/i, g), then ^ih + l^d > tJ-h+tJ-g > 2k+g~h > 2k + d—h. 
This shows that tp satisfies W{h,d) and (a) follows. If ■0 meets (v) and (h,g) ^ D, 
then part (a) implies that D has an outer corner in column g, so V' meets (iii) or 
(iv). This contradiction proves (b). □ 

The following result implies that no term meeting (iv) survives the algorithm, 
and also that applications of (iv) happen in an uninterrupted sequence. 

Lemma 4.6. Assume that ijj meets (iv) and let ijj' = (I?', /i', S' , h) be any successor 
of ip of level h . 

(a) We have {h, g - 1) E D. 

(b) // {h,g) ^ D' then ip' meets (iii) or (iv). 

(c) // (ft., g) e D' and tp' does not meet (v) , then S' = S , the child (D', ^, S,h — 1) 
of ^' meets (y), and gn-i — <7- 

Proof. Part (a) follows from Lemma [4.51 If ip satisfies condition X then assertions 
(b) and (c) follow from Lemma 14.41 so we may assume that X fails and W(/i, g) 
holds for -ip. Without loss of generality we can also replace ip with the most recent 
predecessor whose parent did not meet (iv). Let («,<?) be the outer corner of D in 
column g. Since ■0 does not meet (iii) we have i < h. Furthermore W{i,g) fails for 
Ip, since otherwise the most recent predecessor of tp of level g meets (i). We obtain 
IJ-h + ^'Q < A/i-i+^g < Xi — h + \ + i + < + fig + l + i- h < 2k+l + g-h. Since 
tp satisfies W{h,g) we deduce that Xi — h + l + i ~ Xh-i = f^h = 2k + l+ g — h — fig. 

If S' 3 S, then fj,'g < fj,g, and Lemma 13.61 implies that /' = g. In addition, we 
have either {h,g) ^ S' or ^'^ > /x^ = Xh-i. Since both possibilities imply that ip' 
satisfies condition X, it follows that assertions (b) and (c) are true for tp' . 

Otherwise, we have S' = S and fi' = fi. Then tp' satisfies W(/i, g) and (b) 
is true. Assume that {h,g) £ D' and tp' does not meet (v). Then X fails for 
tp' , so we must have A/i_2 ^ 1 = Xh-i = /ih < fiu-i- Since {h — \,g) ^ S, we 
deduce that (£)', fi, S,h— 1) satisfies condition X. Furthermore, since {h—l,g) ^ C 
and Xh-2 = Xh-i + 1, we obtain {h — 2,g) ^ C, so gh-i = g. We conclude that 
{D' , iJ,,S,h — 1) meets (v), which completes the proof of (c). □ 

Definition 4.7. Let d^C = e dC \ {i, j - I) e C or i = j = m}. 

Corollary 4.8. Let (z D \C, and if 2 < h < m then assume that j ^ g. If 

G d^C then this pair tvas added to D by rule (i), and otherwise it was added 
by rule (iii). 

Proof. Let tp' — {D' , fi' , S' , h') be the most recent predecessor of -0 for which (i, j) ^ 
D' . Then (i, j) is an outer corner of D' . If tp" — {D" , fi" , S", h") is any predecessor 
of tp' meeting (iv), then Lemma 14.61 implies that that h" — h', g^' = g, and 
{h',g — 1) S D', so we must have j — g, a contradiction. It follows that no 
predecessor of tp' meets (iv). If tp' meets (i) then h' = j, and since D' \C contains 
no pairs in column j — 1 we deduce that G d^C. Finally assume that tp' 

meets (iii) and {i,j) G d^C. Let tp = (D,S,'jl,j) be the most recent predecessor 
of tp' of level j. Then (i,j) = {h',j) is an outer corner of D, since otherwise 



A GIAMBELLI FORMULA FOR ISOTROPIC GRASSMANNIANS 



19 



{h' — E D' \ D was added to a 4-tuple on the path from ip to ip' , which is 
impossible. But then the inequahty Jli + Jlj = /i^ + /i^ > 2k + j — i imphes that ip 
meets (i). This contradiction finishes the proof. □ 

We now prove some results that will be used later to show that surviving 4-tuples 
= (D, S, 0) G *o satisfy A ^. 

Lemma 4.9. Let ijj — (Z?, fi, S, h) and let j < £ be a positive integer. 

(a) If h > 1 and {h,j) ^ C and (h + 1, j) G D, then we have fij > Xj. 

(b) If h < 1 or (h — 1, j) G C, then we have fij > Xj — 1- Moreover, if iij = Xj — I, 
then D \ C contains exactly one pair in column j , and this pair is also in S. 

Proof. Suppose that /jj < Xj and choose i > h maximal such that {i,j) G D. 
Let tp' — {D' , n' ,S' ,h') be the most recent predecessor of -0 with {i,j) ^ D' . 
Then -0' meets rule (i), (iii), or (iv), which adds the pair to D' . Let if) = 
{D' U {i,j),'p,S,h') be the child of ip' that is a predecessor of ip- Notice that 
fJ-'t ^ 'Pt ^ ^t-i for ah integers t such that h < t < h' and {t,t) G -D' U 
since otherwise condition X holds for every successor of ip of level t. We also have 
l^-'j < Xj, and if /i^ = A^- then {i,i) € S, i < j, JI^ > ^-j and Jlj < fi'j. 

If ^p' meets (i), then h' = j. Since {i — 1, j) ^ C we have /i- + ii'j < A^-i + A^ < 
2k + j ~ i + 1. As W{i,j) holds for tp' , it follows that /Lt^ = Ai_i and ^'j = Xj. But 
this implies that /Zj > Ai_i, a contradiction. 

Therefore ip' meets (iii) with h' = i, or it meets (iv) with h' > i. In either case 
we have g' = j, and since ip' does not satisfy condition X, it must satisfy W{h' 
Since {h' - ^ C and thus + pt'j < Xh'-i + A^ < 2k + j -h' + 1, it follows that 
M/i' = Xfi'-i and Jlj < ^'j — Xj. We obtain Xh'-i + 'Pj ^'^k + j — h', so Lemma |3^ 
shows that / = j. Since JI^ > //■, we must also have i < h' , so {h' , f) ^ S and tp 
satisfies condition X. This contradiction completes the proof of part (a). 

If fij < Aj — 2, then D \ C contains at least two pairs in column j, say (a + 1, j) 
and {a,j), and the assumptions in (b) imply that a > h. Let ip' = {D' , fi' , S' , a) 
be the most recent predecessor of ^p of level a. Part (a) applied to ip' implies that 
fi'j > Xj, a contradiction since fi'j = fJ-j. □ 

Corollary 4.10. Assume that ip meets (iii) and let {h,j) be the outer corner of D 
in row h. If fij > fJ.j-i, then fij^i = /ij — 1 and D U {h,j) has no outer corner in 
column j . 

Proof. Lemma 14.21 implies that /ij-i < fij < Aj-i. Since {h — 1, j — 1) G C, it 
follows from Lemma l4.9f b) that /jj — Xj-i = /ij-i + 1 and D \ C contains a unique 
pair (i,j — 1) in column j — 1. Since i < j — 1, it is enough to show that i = h. 
Lemma implies that (i, j — 1) was added by (i) or (iii), so ip satisfies W(i, J — 1), 
and we obtain fii + fij = fii + fJ-j-i -\- 1 > 2k + j — i. Assume that i > h and let 
Ip' = {D', ji' , S', i) be the most recent predecessor of tp of level i. Then (i, j — 1) G D' 
and g' — j. Since fi'^ = fii and fx'j > iij, W{i,j) holds for ip' . But then ip' meets 
(iv) and is not the most recent predecessor, a contradiction. □ 

Lemma 4.11. Assume that j > m. If h ~ or {h,j) G D, then fij < /ij-i- 

Proof. Assume that /ij > fJ-j-i- Then Lemmas 14.21 and I4.9f b) imply that fij = 
Xj-i = fij-i + 1, and D \ C contains a unique pair — 1) in column j — 1, 
with i > h. Let ip' = {D' , fi' , S' , i) be the most recent predecessor of ip of level 
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i for which ^ D' . The assumptions of the lemma then imply that ip' ^ ip. 

Lemma [4.61 shows that — 1) was added to D by (i) or (iii), so ip' satisfies 
W{i,j — 1). Since fj,j > fij > Hj-i = Mj-u V'' also satisfies W(i,j), so ip' meets 
(iii) or (iv). The choice of tp' implies that (i, j) must be the outer corner added to 
D\ so in fact V'' meets (iii). Now the statement of rule (iii) implies that (i, j) G 5, 
so ^'j > > Mj-i- This contradicts Corollary 14. 101 □ 

Lemma 4.12. Assume h < m. Then fj,j < Xj-i and iij < /ij-i for h < j < m. 

Proof. If the statement is false, then choose i > h minimal such that fii > Xi^i or 
A*i > A*i-i, and let ^j/ = {D' , fi' , S' , i) be the most recent predecessor of of level 
i. Since /x- > Ai_i or ^- > ^-^^ and condition X fails for ^' , we have ^ D', so 
i = m and /ij„ > A„i-i > k. But then /ij + > At + > {k + m — t) + k for 
1 < t < m, so ip' meets (i), a contradiction. □ 

Lemma 4.13. Assume that {h,h) G D, fi^ = ^h-i, and [h,Xh-i] G R. Then ip 
satisfies condition X and does not survive the algorithm. 

Proof. Since [h, Xh-i] G R and r{h + Xh-i) = g, we have < 2k + g — h — Xh-i, 
hence W(/i, g) fails for tp. Lemma l4.12l shows that iJh > M/i+i > • • • > Mmj so W((i, g) 
fails for ft, < d < TO. If (ft, g) ^ D then condition X holds because (ft., /) = (ft-, g) ^ S. 
Otherwise choose i > h maximal such that {i, g) e D, and let tp' = (D', /i', 5", ft') be 
the most recent predecessor of ^p for which {i,g) ^ D' . Then meets (iv) and ip is 
a successor of the child ip" = {D, fx, S, ft') of ^p'. If ft' > ft, then Lemma [4.41 implies 
that X fails and W(ft',.g) holds for ip' . Since W(ft',g) fails for -0", it follows that 
i < h' and /x/j/ + /ig = 2fc + g — ft'. We also have A/i/_i + /ig < A/i_i + fig + h — h' < 
2k + g — h', so fih' > X^'-i, and using Lemma [3?^ we get {h',f") — {h',g) ^ S. But 
then ip" satisfies condition X, a contradiction. We therefore have ft' = ft, W(ft, g) 
fails for ip' , X holds for ip' , and the result follows from Lemma [4.41 □ 

In our applications of Lemma 14. 131 we only need the fact that "0 does not survive 
the algorithm, so it is enough to know that a predecessor of ip meets (iv). The last 
six lines of the above proof could therefore be omitted. 

4.3. In this section we will study a 4-tuples — {D, fi, 5, 0) £ with fii+i > 0. 
For such a 4-tuple, Corollarv 14.81 implies that each pair (i,j) G D \ C was added 
by (i) or (iii). More precisely, the pair (i,j) was added by (i) if {i,j) G d^C, and 
otherwise the pair was added by (iii). 

Proposition 4.14. Suppose that tp = (Z?, /i, 5, 0) and fie+i > 0. Then fi is a k- 
strict partition with = |A| + p, satisfying Xj — I < /ij < Aj^i for every j > 1, 
and Xj < fij when Xj > k. Furthermore, we have D = C^+i(/i). 

Proof. By Lemma 14.121 we have fij < Aj_i and fij < fJ-j-i for 1 < j < to, and 
Lemmas 14.21 and 14.111 show that p,j < min(Aj_i, /ij_i) for j > m. We deduce that 
/X is a fc-strict partition. Lemma l4.9f b) implies that A^ — 1 < ^.j for every j. Clearly 
Xj < pLj when Xj > k, and = |A| + p. 

It remains to show that the set Ce+i{p,) is equal to D. li D C(»+i(/i), then 
since Ce+i{fJ-) and D are both valid sets of pairs, we can find an inner corner 
(j, j) G £> \ Ci+i{fi) such that {i + 1, j) ^ D and {i,j + 1) ^ D. Since {i,j) ^ C by 
Lemma [^m the pair {i,j) was added by (i) or (iii), and W(i,j) holds since fii and 
Hj did not change since this event. 
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On the other hand, if Ce+i{p) <f_ D, then we can find an outer corner («, j) of D 
such that (i, j) G Ce+i{ii). If (i, j) = (m, m) or if (i, j — 1) G C, then the most recent 
predecessor of ^ of level j meets (i), and otherwise we deduce from Lemma l2.1l that 
the most recent predecessor of tp of level i meets (iii) . This contradiction finishes 
the proof. □ 

Lemma 4.15. Assume that ip — [D, fj,, S, 0) and j are such that fij = Aj — 1 > 0, 
and let be the unique pair in column j of D \ C. Then the removed box [j, Xj] 
and the above box [j — l,Xj] are k-related to the boxes [«,c] and [i,c—l], respectively, 
where c=2k + 2+ j — i — Xj, and these latter boxes belong to R. 

Proof. Let ijj' = {D' , /i', S' , h') be the most recent predecessor of i/j for which (i, j) ^ 
D' . Since fi'j = Xj and ip' satisfies W(i,j), we obtain fJ.i > fi'i + 1 > c, and since 
{i,j) ^ C we similarly have Xi < c — 2. The boxes [i,c] and [i,c — 1] belong to R 
because fij+i < Xj and ^j < Xj (see i )3.2l) . □ 

Lemma 4.16. Assume that tp — {D, fi, S,0), fJ-e+i > 0, and let i < m be any 
integer such that fii = A^-i. Then [i, Ai_i] ^ R and {i, fi{fJ.)) G S. 

Proof. Let ip' — {D' , /i', S' , i) be the most recent predecessor of ip of level i. Then 
/i^ — fii and («, i) G if i = TO this follows because /i^ = Am-i > k. Lemma [4. 131 
implies that [i, Ai_i] ^ R' . Since all pairs (c, d) G I? \ £>' were added by (iii) and 
satisfy d > g' , it follows that fi'j = jjLj for m < j < g', so Lemma [3.71 shows that 
[i,Ai_i] ^ R and /' — fiifJ.). Finally, we must have {i, f) G S" C 5 since i/j' does 
not satisfy condition X. □ 

Proposition 4.17. If tp = (D,/i, S*, 0) and ni+i > 0, then we have A — ?> /i. 

Proof. By Proposition 14.141 it suffices to check that conditions (1) and (2) of §2.21 
are true. Condition (1) follows from Lemma 14.161 since [i, A^^i] ^ R for each i. 
Suppose that fij + 1 ^ Xj = d for ji < j < j2 • According to Lemma I4.15[ each 
removed box [j, d] for ji < j < j2 is fc-related to some box [ij, Cj] G \ A, and the 
box [ij, Cj — 1] is also in /i \ A. Condition (1) implies that each box [j, d] is /c-related 
to at most one box oi fi\ X. It follows that if j < j2, then [ij, Cj] = [ij+i, Cj+i — 1], 
so all the boxes [ij, Cj] lie in the same row of /i \ A. Condition (2) follows from this 
since we also know that the box [ji — l,d] is fc-related to [ij-^, Cj-^ — 1]- D 

4.4. Propositions l4T4l and l4T7l tell us that if ip ^ {D, fi, S,0) is any 4-tuple in ^E'o 
with fie+i > 0, then is a fc-strict partition with X ^ fi, D — Ci-i-i{fi) is uniquely 
determined by /z, and ev(tp) = T{C{n), ji) is a term appearing in the Pieri rule ([T4l) . 
To account for the multiplicities, we give an explicit construction of the possible 
sets S in these 4-tuples. In this section we fix an arbitrary fc-strict partition )i such 
that X ^ pL and \ii[ = |A| +p. 

A component means an (edge or vertex) connected component of the set A of 
§2.21 We say that a box i? of A is distinguished if the box directly to the left of 
B does not lie in A. We say that B is optional if it is the rightmost distinguished 
box in its component. Notice that ^Tt(A, /x) is equal to the number of optional 
distinguished boxes in A. To each distinguished box B = [i,c\ we associate the 
pair {i,j) = (i,r{i + c)). The inequality Xi-i > 2k + i — {i + c) implies that i < j, 
so {i,j) G A. Let E (respectively F) be the set of pairs associated to optional 
(respectively non-optional) distinguished boxes. We furthermore let G be the set 
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of all pairs (i, j) £ A for which some box in row « of /x \ A is fc-related to a box in 
row J of A \ /i. 

Lemma 4.18. (a) We have E (J F U G C Ce+iifJ,) n dC. 

(b) Each pair in E (J F is associated to exactly one distinguished box of A. 

(c) The sets E, F , and G are pairwise disjoint. 

(d) If{i,j)eF, thenj^Mfi). 

(e) // {i,j)eEL)F(JG,i< j, and {i,j - 1) ^C, then fij < Aj_i. 

Proof. Let G G. Then fij — Xj ~ 1 and the boxes [j, Aj] and [j ~ l,Aj] 

are fc-related to [i,d] and [i,d — 1], where d = 2k + 2 + j — i — Xj. We also 
have Xi + 1 < d < ^i. Therefore A, + Aj < d + Xj — 1 = 2k + 1 + j — i and 
/^i + Aij > d + jij = 2fc + 1 + j — i, so {i,j) G Cf+i(/i) \ C. Assume that {i,j) is 
associated to a distinguished box [«, c] G A. Since r(i + c) = j < j + 1 = r{i + d), 
we must have c < d, hence fj,j = 2k + l+ j — i — d<2k + j — i — c. But then 
[i, c] G a contradiction. It follows that G r\{E U F) = 9. 

Now let [i, c] G A be distinguished and set j = r{i + c). Then j > r{i + Xi + 1), 
so (fT6|) shows that ^ C. Since [i,c] ^ we get > 2k + j — i — c > 

2k + j — i — fii, hence {i,j) G Ce+i{fi). Using Lemma [2.11 this establishes (a). If 
[i, c'] G R{p-) is any box with c' > c, then we must have j < r{i + c'), since otherwise 
fij <2k+j~i — c'<2k + j — i — c. This proves (b) and finishes the proof of (c). 
If {i,j) G F, then fii — Ai_i, c = ei{fi), and j = fiifJ-)- This establishes part (d). 

In the situation of (e), notice that if /ij — Xj^i, then {i,j) G E'UF is associated 
to a distinguished box [i, c] G A. We must have c > fc + 1, since otherwise A^ < k, 
i = m, and j — r{m + fc + 1) = i. Since [i,c\ ^ R{fJ-) and (i, j — 1) ^ C, we also have 
c > 2k + j -i- = 2k+ j -i- Aj_i > A^ + 1. It follows that [i,c- 1] G Set 
j' = r{i + c — 1). Then j' < j and /ijv — j' < 2fc — « — c + 1 < /ij — j, which shows 
that j' — j and /ij = 2fc + — i — c + 1 < Aj_i. This contradiction proves (e). □ 

To every subset E' of E we associate the set of pairs S{E') :— E' U FUG. This 
is a disjoint union, and there are exactly 2'''^*^'^''^'' sets of this form. The following 
proposition therefore completes the proof of Claim 1. 

Proposition 4.19. Let S C A be any subset. Then (C£+i(/Lt), /z, 5, 0) G V&o if and 

only if S — S{E') for some subset E' C E. 

Proof We first assume that ip = (C£+i(/x), ^, S", 0) G *o- Lemmas lLT6l and l4T8l d) 
then imply that F C S. We next show that G = {{i,j) & S \ fij < Xj}. If /ij < Xj 
then S contains a unique pair {i,j) in column j. Lemma 14.151 shows that [j, A^] is 
fc-related to a box in row i oi fj, \ X, and condition (1) implies that no other box 
in /Lt \ A is fc-related to [j, Xj]. It follows that («, j) is also the unique pair of G in 
column j . 

Let (i, j) G S\G. We wiU show that («, j) is the pair associated to a distinguished 
box of A. If i = J = m, then A„i < fc < fim and (m, m) is associated to the 
distinguished box [to, fc + 1] G A. We can therefore assume that i < j, hence 
fii > Xi. Since {i,j) ^ G we also have A^ < fj,j. If A, -I- fij > 2k + j — i, then the 
inequality Xj-i > Hj >2k+j — i~Xi~l implies that j < r{i -\- Xi + 1). Since 
(ij) ^ C, it follows from (HH) that j = r{i + Aj + 1). We deduce that [i, A,, + 1] G A 
is a distinguished box and is the associated pair. 

Otherwise we have Xi + fij < 2k + j — i. In this case we set c — 2k+j~i~fij. 
Since (i,j) G C£+i(/i) we have Ai < c < /ii. We also have c > fc; if z = to this 
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follows because jij < Am < k. We claim that fij < Xj-i- If — 1) G C, then this 
follows because Hj < 2k + j — i — Xi < Xj-i, so assume that — 1) C. Then 
we must have j > m, and was added to S in Phase 2 of the algorithm. By 
Lemma [4?2l the first predecessor (£)', /i', S", i) of i/j of level i satisfies that < Aj_i . 
Since ^ 5", this implies that fXj < Aj_i, as claimed. 

The inequality Aj_i > fij = 2k + j~i — c implies that r{i + c) > j, and since 
Aj < we also have r(i + c + 1) < j. We deduce that r{i + c) = r{i + c + 1) = j, 
[i,c] e R{lJ-), and [z, c + 1] G A. This shows that [i, c + 1] is distinguished and {i,j) 
is the associated pair. We conclude that the set E' :— S \ {F U G) is a subset of 
E, hence S — S{E') has the required form. 

Now let E' C E he an arbitrary subset and set S = S{E'). We must show that 
(C£+i(/i), /i, 5, 0) e ^'o- Set ly = Y[{i j)£S ^^^^ definition of S ensures that 

^ > A, so G J\f{X,p). We now construct a path V in the substitution forest by 
applying the substitution rule of §3.3l repeatedlv to the initial 4-tuple {C,i',9,£+1). 
Whenever the substitution rule assigns two children to a 4-tuple tp' oi V, we use 
the set iS* to determine which child is to follow ^p' on the path. More precisely, 
if tp' — {D' , n' , S' , h') meets (i) or (iii) and has two children, and if is the 
outer corner being added to D' , then we choose the child V" = {D" , fi" , S" , h') 
for which S" \ S' = S Ci {(«, j)}- We will show that V terminates in the 4-tuple 
{Ci+i{ti),fi,S, 0). 

For /i > we set Dh = CU{{i,j) G Cg+iilJ.) \ i > h or {j > h and (i, G C)}. 

Lemma 12.11 implies that this is a valid set of pairs. We will say that the 4-tuple 
tJj' = (£)', ^',5", h') is good if it satisfies L»v C D' C Ai'-i and S' =^ S f] D' . It is 
enough to show that if -0' is any good 4-tuple on V with h' > 0, then tp' has a good 
child that also belongs to V. 

Let be a good 4-tuple of P. We then have ^ = 11(1 j)e5-^D' A^'- 
first show that if tp' meets (i) or (iii), and is the pair being added to D', 

then is also in Dh'-i- If + /ij — + i^'j then this is true because ip' 

satisfies W(i, j), and otherwise S \ D' must contain at least one pair in row i or 
column j, which implies that G -D/i'-i by Lemma |4T8l[a). On the other hand, 
assume that D' C Dh'-i- Then Dh'-i \ Dh' contains an outer corner of 
D' . If we choose c > j maximal such that (i,c) G C£+i(yu), then the inequalities 
fi'i + ^.'j > Hi + ~ {c — j) > Hi + fie — c + j > 2k + j — i show that ip' satisfies 
W{i,j). If h' — j then ip' meets (i), and otherwise we have h' = i < j and ip' 
meets (iii). Notice also that if -0' meets (iii) and fij > ^'j-n then we must have 
{h',j) G S since is a partition. These observations show that tp' meets (i) or (iii) 
if and only if D' C Dh'-i, and in this case tp' is succeeded on P by a good child. 

Now consider a good 4-tuple tp' of V such that D' = Dh'-i- It remains to 
show that the substitution rule simply decreases the level of tp' , i.e. tp' does not 
meet (ii), (iv), or (v). Assume that tp' meets (ii) and choose i>l minimal such 
that {i,h') ^ D' . Then ^'h' > -^/I'-i and {i,h') is not an outer corner of D' . We 
have i < h' and {i,h' — 1) C. Using Lemma [2.11 we deduce that {i,h') G S and 
Hh' — Xh'-i', however this contradicts Lemma k.lSf e). 

If tp' satisfies condition X, then h' < m and /ij^, = /i^' = Xh'-i- It follows that 
A contains a non-optional distinguished box in row h' , and Lemma k.lSf d) implies 
that {h', fh'in)) G F is the associated pair. But Lemma [3771 shows that /' = fh'ilj), 
so {h' , /') G Sr\Dh'-i = S' . We conclude that condition X fails for tp' . In particular, 
tp' does not meet (v). Finally, if tp' meets (iv), then {h'.g') ^ C£+i(/i), and since 
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Figure 5. (i) A and n, (ii) C(A) and C^+i(^) in Example 14.201 



li'i-^, = iih' and /ig, = i-ig', we deduce that W{h',g') fails for ijj' . This contradiction 
finishes the proof that the level of tp' is decreased. □ 

Example 4.20. Consider the partitions A = (22,21,18,16,14,7,5,4,3,3,1) and 
fi = (25, 21, 19, 17, 15, 14, 6, 5, 3, 2, 2), and set fc = 5. Then A /x. The diagrams 
of A and /i are displayed in Figure O with boxes from R{fi) labeled with R and 
distinguished boxes labeled with O for optional and N for non-optional. The figure 
also shows the pairs in C(A) and C^+i(/i), where £ — 11. The pairs from E, F, and G 
are labeled accordingly, and the one additional pair from C£+i(/i) \ C(A) is labeled 
D. The skew dotted lines help to identify the pairs in E and F associated to the 
distinguished boxes. The compositions for which some 4-tuple (Cf+i(/i), /i, 5, 0) 
originates from (C,i^, 0,^ + 1) may or may not include the boxes labelled with 
question marks, which can be traded for boxes from the rows of corresponding 
optional distinguished boxes. There are 2^ such compositions v, and for each of 
them there are two sets S, one of which contains the diagonal pair (7, 7) G E. 



Remark 4.21. The most subtle ingredient of the Substitution Rule is the reference 
to condition W{h,g) in rule (iv). In fact, if we modify the algorithm so that 4- 
tuples can meet (iv) only when they satisfy condition X, then Claim 1 still holds 
but Claim 2 fails. To see this, let tp = {D, fi, S, h) be a 4-tuple that meets (iv) but 
does not satisfy condition X, and assume that the parent of ip does not meet (iv) . 
Then the modified algorithm replaces "0 with (D^fj,,S,h — 1), and the arguments 
in the proofs of Lemmas 14.41 and 14.61 can be used to show that the latter 4-tuple 
satisfies X and does not survive the modified algorithm. It follows from this that 
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Claim 1 is true. However, if the modified algorithm is applied with A ~ (4, 3, 1, 1), 
p — 6, and k = 1, then the resulting set of 4-tuples meeting (ii) or (v) contains 
1119 elements, and 543 of these have non-zero evaluations. This implies that there 
is no involution i : — such that ev{tp) + ev(t('0)) = for every £ ^i- 

4.5. In this section we construct a sign-reversing involution t : ^E*! — > ^'i and show 
that it has the properties required by Claim 2. 

Given a valid 4-tuple tp — (D, fi, h) with h > 2 and fj,h > A^_i, we define 
a new 4-tuple Lip as follows. If {h,h) ^ D, then set tip = {D,J1, S, h), where the 
composition Jl is defined by Jlh~i = M/i ~ 1; M/i — l^h-i + 1, and /it — /it for 
t ^ {h ~ 1,^}. If {h,h) G D and fj-h-i = ^J■h, then set iip — ip. Assume that 
{h, h) € D and /i^-i 7^ /ift- Let zu be the involution of A that exchanges (h — 1, g) 
with (/i, /), and fixes all other pairs. Then set up = {D, /I, S, h), where S = tu{S), 
and Jl is the composition obtained from /i by switching the parts fih-i and fih- 

Lemma 4.22. If ip ^i, then i{L%p) = i/j and ev{ip) + ev{Lijj) = 0. 

Proof. Assume that {h, h) ^ D. Then i.{iip) = ip is clear, and since ip meets (ii), it 
follows from Lemma [1.21 that ev{ip) + ev{Lip) = 0. Next, assume that {h,h) S D. 
Then ip meets (v) and Lemma 14.51 shows that {h, g) G D. It follows that S C D 
and Lijj is a valid 4-tuple. Lemma 13.71 implies that the same values of e, /, and g are 
assigned to "0 and Lip, so we have i(tV-') = '4'- Finally, since ip satisfies condition X, 
we have fih > ^h-i > k, so it follows from Lemma 11.31 that ev{ip) +ev{Lip) =0. □ 

Given any valid 4-tuple ip = {D, fj,, S,h), a valid set of pairs D' C D, and an 
integer h' > /i, we define the 4-tuple ip{D' , h') = (D' , JJ^. j-^^s-^D' ^ ^ ^')- 
Notice that if ip occurs in the substitution forest, then every predecessor of ip can 
be written as ip{D' , h'). In particular, the initial 4-tuple leading to ip is ip{C,£+ 1). 

To finish the proof of Claim 2, we will show that C ^i. Fix an element 

Ip — {D, /i, S, h) G We will show that the substitution forest has a path leading 
to Lip and that this 4-tuple meets (ii) or (v). Define compositions v and v by 
{C,iy,(/},e+l) = ip{C,£ + l) and {C,v,9j+1) = Lip{C,e+l). 

The following lemma shows that t?/" G ^'i whenever ip meets (ii). We will say 
that two valid 4-tuples meet the same rule, if both meet the same rule among (i)- 
(v) in the substitution rule, or if the substitution rule decreases the level of both 
4-tuples. 

Lemma 4.23. Assume that G meets (ii). 

(a) We have v G Af{X,p). 

(b) Let Ip' = ijj{D\h') be any predecessor of ip. Then Lijj(D',h') meets the same 
rule as ij/ . In particular, Lip meets (ii). 

Proof. Notice first that h > as D has no outer corner in column h and (h, h) ^ D. 
Part (a) is true because Vh-i = Jih-i = /^h - 1 > hi-i-, i^h > Jj-h ^ IJ-h-i + 1 > 
A/i-i > Xh, and vt = vt > At for t ^ {h — I, h}. The inequality Jlh > X^-i implies 
that Lip meets (ii). Let ip{D' , h') be a strict predecessor of ip. li h' > h and (i, h') 
is an outer corner of D', then i<m~l<h — 1 and W(i, h') holds for Lip{D' , h') 
if and only if it holds for ip{D',h'). Part (b) follows from this when h' > h, and 
when h' = h \t follows from Lemma l42l □ 

From now on we assume that ip meets (v) and that ^t-i 7^ /^/i, so that Lip ^ ip. 
We have {h,h) £ D, ip satisfies condition X, and /i/j > Xh-i. We also have fih = 
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fih-i > A/i-i, and in case of equality /i/i-i = Xh-i we must have {h — 1, g) ^ S or 
equivalently {h, /) ^ S. This shows that Lip satisfies condition X. 

Lemma 4.24. We have i/ e Af{X,p) . 

Proof. Notice that ly > X and = z^i for i ^ {h — l,h,f,g}. Observe also that 
row h — 1 oi D \ C contains the single pair (h — 1, g). If ]lh-i > A/i_i we therefore 
obtain Vh-i > Mh-i — 1 > A/i_i. Otherwise we have Hh = J^h-i — Xh-i, and since 
Hh~i 7^ M/i and V' satisfies condition X, it follows that {h,f) ^ S, so {h~ l,g) ^ S 
and ^Ti-i = A^_i. Using Lemma 13.51 we also obtain i/h ^ Jj-h ^ [g ^ b + I) > Xh- 
Notice that li h < f = g, then Vg — Vg > Xg, so we may assume that / < g. 
Lemma 14.31 then implies that Vg > Jig — /.ig — Xg-i > Xg. Using that [h,e\ ^ R 
and the definition of /, we obtain fif > 2fc + / + l — /i — e > A/. If / > h then 
we also have i/f > Jlf; when h < m ^ f this is true because {h, f) ^ C implies 
{f,f + ^)^D. We conclude that T'f >Jif ^ fJ-f > A/, as required. □ 

For i G N we define the vahd set of pairs Dt = C U & D \ i < t}. Let 

z > 1 be the smallest positive integer for which {z,g) ^ C and ■ip{Dz,g) does not 
meet (i), and choose z > 1 minimal such that {z,g) ^ C and LiplDzTg) does not 
meet (i). We also write zi = min(z,F) and Z2 = max(z,z), and define the sets 
F = {(«, j) ^D\C\j<g} and G — {{i,g) \ zi < i < Z2}. Let ho be maximal 
such that {hQ,g) £ D. Notice that h < ho, D^^ = D^^ U G, and G C d^C. 

Lemma 4.25. (a) Both ijj[Dz,g) and ip{Dz U F,ho) are predecessors ofip. 
(b) We have Z2 < ho + 1. 

Proof. All pairs G I? \ C with i < z were added to D in Phase 1 of the 

algorithm, while all pairs (i, g) G D\C with i > z were added in Phase 2. Lcmma [4.6l 
implies that the latter pairs were added to predecessors of tp of level ho, and this 
happened after all pairs of F were added. Part (a) follows from this. 

For (b) we may assume that {ho + 1, g) G 9^C. Since Jlho+i = t'-ho+i Jig — pg, 
it is enough to show that W(/io + 1, <?) fails for ip. li ho + 1 < m, then this follows 
because the most recent predecessor of -0 of level ho + 1 does not meet (iii) or (iv). 
Otherwise we have ho + I ~ g ~ m, in which case the inequality /i„j < k follows 
because W(z,m) fails for ip{Dz,m). □ 

Given two valid 4-tuples ip' and ip" , we write ip' < tp" if ip' is a predecessor of 
ip", with iP' = V" allowed. We wiU write ip' < ip" if V'' < V and V'' 7^ V- The 
next proposition implies that iip appears in the substitution forest; in fact we have 
Lip{C, £ + 1) < iip{DT;, g) < Lip{D^ U F, ho) < Lip. 

Proposition 4.26. Let ip' — ip{D' , h') he a predecessor of ip. 

(a) If Ip' < ip{Dz^,g) then Lip(D',h') meets the same rule as ip' . 

(b) If zi <t<z, then iip{Dt,g) meets (i). 

(c) If ip{Dz, g) < Ip' < ip{Dz U F,ho), then Lip{D' AG,h') meets the same rule as 
Ip'. Here D' AG = {D' UG) \ [D' n G) is the symmetric difference. 

(d) Ifz<t< ho, then Lip{Dt U F, ho) meets (iii) or (iv). 

(e) If ip{D, ho) < Ip' , then Lip{D' , h') meets the same rule as ip' . 



Example 4.27. Let A = (3,1), p = 4, and /c = 1. Then C = {11,12}, and 
the 4-tuple ip = (£>, 341, {22}, 2) e fi meets (v), where D = {11,12,13,22,23}. 
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We compute f — 2, g — 3, and Lip = (_D, 431, {13}, 2). We also have z — 1, 
z — 2, ho — 2, F ^ {22}, and G = {13}. The paths of the substitution forest 
leading to tp and iip are displayed below. Notice how the path from il;{Dz,g) to 
ip(Dz U Fjho) translates to a path from l4>{Dz, g) to i4>{Dz U i^, /iq), and the path 
from ■il!{DzUF, ho) to tp(DzL>F, ho) translates to a path from Ltp{Dz,g) to tipiDz, g). 



i>{D^,g) = ({11, 12}, 341, 0,3) 

\^ 

({11, 12}, 341, 0,2) 

(i) 

T 

i)(D^UF,ho) = ({11, 12, 22}, 341, {22}, 2) 

(iv) 

^{D^UF.ho) = ({11, 12, 13, 22}, 341, {22}, 2) 

(iii) 

i> = ({11, 12, 13, 22, 23}, 341, {22}, 2) 



V 



({11, 12}, 332, 0,3) = LJp{D^,g) 

(i) 

\/ 

V 

({11, 12, 13}, 431, {13}, 3) = L4>(D;,g) 
\/ 

({11, 12, 13}, 431, {13}, 2) 

(i) 

X y 
V 

({11, 12, 13, 22}, 431, {13}, 2) = i.-ip{D^UF,ho) 
(iii) 

({11, 12, 13, 22, 23}, 431, {13}, 2) = l^j 



The proof of Proposition 14.261 is based on the following lemmas. 

Lemma 4.28. If ho < h' < m and gu' — g, then Jlh' = Hh' < ^h'-i- 

Proof. Let tp' ~ {D' , fi' , S' , h') be the most recent predecessor of ip of level h', 
and notice that Jlh' = /i/i' — n'j^,- If {h',h') ^ D' then h' = m, and since ip' 
does not meet (i) we obtain /i^ < k < Xm-i- We may therefore assume that 
{h',h') S D'. Since condition X fails for tp' by Lemma WM we have < Xh'-i- 
Suppose that fi'/^, = Xh'-i- Then {h' , fh'{fJ.')) S S', and since {h',g) ^ D we 
must have fh'ifJ-') < 9 = 9h'- Lemma 14.31 then implies that /i^ — Ag_i, and since 
(/i' — 1,(7 — 1) e C we deduce that W{h',g) holds for ip' . But then Lemma [4.5f a) 
implies that ip' meets (iii) or (iv), contradicting the choice of ip' . We conclude 
that n'l^, < Xh'-i, as required. □ 

In the next three lemmas, we let ip{D' , h') — {D' , fi' , S' , h') denote a predecessor 
of Ip, let Z? be a valid set of pairs such that D' \ G C D C D' IJ G, and write 
Lip(D,h') = (D,-p,S,h'). 

Lemma 4.29. Choose {i,j) & dC\ D with j ^ g. Assume that (a) — 1) e C or 
i > h, and (i,j) is an outer corner of D, or (b) {i,j) = {h',gh>) ^ {m,m). Then 
ip{D',h') satisfies W{i, j) if and only if Lip {D,h') satisfies W{i,j). 

Proof. Notice first that i < g, since otherwise i — m ~ g < j and {i,j) ^ dC. 
We also have j > Z2, as otherwise Z2 — 1 = to = j, {m,g) is an outer corner 
of Dm, and g = m. Furthermore, if zi < i < Z2, then we obtain {i,g) € d^C, 
j > g, i < h, and {i,j — 1) G C, which is a contradiction. This shows that 
hj ^ {zi,zi + 1, . . . , Z2 — 1,5}, so condition W(i, j) holds for Ltp{D, h') if and only 
if it holds for Lip{D' , h'). Henceforth we will work with the latter 4-tuple, which we 
denote Li;{D',h') = {D' ,J1' ,S' ,h'). We then have n't = Jl't for t i {h-l,h,f,g}. 
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Assume first that i = h. Then we have m < j < g, and {h,j) is an outer corner 
of D'. Since {h,j) e D it follows that W{h,j) holds for %l:{D',h'). We must prove 
that also tilj{D' , h') satisfies W(/i, j). If j = h, then we have i ^ j = h ~ ni, and 
since {h — 1, g — 1) G C we obtain 

Ji'h > V-h ~ g + h = fih-i - g + h> \h-i - g + h> 2k~ Ag-i > k . 

We can therefore assume that i = h < j. We have /i^ = Dj > Xj, and since Lip 
satisfies condition X we also have pt'^+ 5— j + 1 >Ji.h> A^-i- Since {h—l,g—\) G C 
we obtain 

V-'h + V-'j > ^h-i + - .g + j - 1 > A/,_i + Ag_i - g + j- l>2k + j- h. 

If W(/i, j) fails for Lip{D', h'), then we must have equality Ji'i^+g—j + l = Jj-h = ^h-i, 
hence {h,t) G S for j <t < g. Since Xt-i < Jih-i 7^ Jj'h we also obtain /x/j < ]lh-i- 
Since Lip satisfies condition X, we deduce that (/i, /) ^ 5*. But then f < g, and 
Lemma l473l implies that {h,g) ^ S and hence {h,g) ^ 5, a contradiction. 

We next show that if i ^ /i, then the identities /i^ = pi^ and /x^- = /I^ hold. If 
i < h, then j > g and (i, j — 1) £ C. Since this implies that i < h — 1, the identities 
are true in this case. Otherwise we have h<i<j<b<f, and the identities are 
clear unless j = /. In this case we have b = f < g. Notice also that {h, /) e D'; in 
case (a) this is true because (i, /) is an outer corner of D' , and in case (b) it follows 
from Corollary 14.81 because h' — i < j — f and {h, /) G I? n d^C. The sets of pairs 
S \ D' and S \ D' therefore agree in column /, and since fif — Jlf by construction, 
we deduce that /i^ = /ij, as required. □ 

Lemma 4.30. The 4-tuple Li/j{D,h') does not meet (ii). 

Proof. Suppose that L%lj{D,h') meets (ii). Without loss of generality, we may also 
assume that i/j{D',h') is the most recent predecessor of ip of level h'. We have 
h' > TO, and since D \ D contains at most one pair in column h', we obtain 
Xh'-i < M/i' ^ 1 ^ J^h' = ^J'h' < Lemma 14.21 therefore implies that /J^' ^ 1 = 
fj,h' ~ fJ^'fii = A/i'_i. In particular, we have h' G {/, 5}- 

Assume that h' — f and {h,f — 1) ^ C. Then [h,Xh-i] ^ -R(Ai), since otherwise 
/ = r{h + Xh-i) and fif < 2k + f — h — Xh-i < A/_i. We also have e > k + 1, 
since otherwise Xh < k^ h = and / = r{m + fc + 1) = to. Since [/i, e] ^ R{p) 
and (/i, / - 1) ^ C, we obtain e>2k + f~h-fif^2k + f- h- A/_i > A/i + 1. 
It follows that [h,e - 1] G Set = r{h + e - 1). Then /i < / and 

fj-fj^ — fi < 2k — h — e + I < fif — f . Since Lemma 14.111 implies that fij < 11 f-^, we 
therefore obtain /i = / and /i/ = 2k + f — h — e + 1 < A/_i, a contradiction. 

In view of the above, the absence of an outer corner in column h' of D implies 
that either h' = f and {h, f) € D, or h' = g and {h — 1, g) G D. It follows that the 
sets of pairs S \ D and S \ D agree in column h' , so Jlf^, < fi'i^,. This contradiction 
finishes the proof. □ 

Lemma 4.31. If h' > ho, or if h' > h and D ~ D' , then L%p{D, h') does not satisfy 
condition X. 

Proof. We first show that /ij^ > Xh and /I^ > A^- In fact. Lemma 13.51 implies that 
^ih = Jlh-i > Xh-i > Xh + g - b + I. If A/i > ^Jj, then we must have Jlh-i = Xh-i 
and {h,j) G S for b < j < g. The equality shows that (h — l,g) ^ S, which in 
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turn implies that {h, f) ^ S, a. contradiction. The inequaUty Jlf^ > Xh is proved by 
interchanging 'ip{D' , h') and Lip{D, h'). 

For proving the lemma we may assume that {h' ,h') G D and /I/j/ > Xh'-i- The 
assumptions imply that D and D' agree in all rows i with i > h'. In particular 
{h'^h') G D' . Since Jih' — iJ-h' and S agrees with S in row h' , we deduce that 
V-h' = A*/i'- Since tp{D' , h') does not satisfy condition X by Lemma [4.41 we obtain 
V-h' = Ai'h' = ^h'-i < fJ-h'-i and {h'Jh-it^')) S S' . We claim that Jly^, < Mv^i- If 
h' — 1 — h, then this is true because Jlf^, — Xh <'Ph- On the other hand, if ft,' — 1 > h 
and the claim fails, then D and D' differ in row h' — 1, and this implies that h' > ho 
and {h' — l,g) G G, hence h' — 1 = ho > h. Since {ho,g) S G C we obtain 
= &fto = 9; so Lemma [4.281 implies that /ij^, < /i/i' < Xh'-i, a contradiction. 

We claim that /Ig, , = A*g^, • Notice that gh' < /, and the claim is clearly true if 
gh' < /. If gh' = f < g, then CoroUarv 14.81 implies that (/i, /) was added to D' by 
rule (i), hence /I^ = fi'j:. li gh' = f = 9, then Lemma [4.281 implies that h' < ho and 
D = D' , so the claim is clear unless {h — l,g) G D' and {h, g) ^ D' . In this case the 
inclusion {h' , fh'{iJ.')) € D' implies that < g- Lemma [4.31 then implies that 

n'g — Xg-i, and since {h,g — \) G C we obtain /i'^ + /Zg > + Ag_i > 2k + g — h. 
Since the outer corner (ft, of D' was not added by (i), it follows that (ft — 1, g) was 
added by (iv) when the substitution rule was applied to the parent of ?/'(£'', ft'). 
By applying Lemma [4.6r c) to the parent of tp{D',h') and using that ft' > ft, we 
obtain {h — 1, g) ^ S and (ft,g) ^ S. This shows that fi'g = /I^, proving the claim. 
Finally, Lemma [3.71 implies that fh'i'P) — fh'ifJ-'), therefore {h',fh'{ji)) G S, and 
we conclude that iip{D, ft') does not satisfy X. □ 

Proof of Proposition\T^ Write tp' = ^p{D' , ft') = (£>', //, S", ft'). 

(a) . If V^' < Tp{D^^,g), then (ft', ft') ^ Z?'. If ft' > g, then Lemmas 1121] and 1130] 
imply that Lip{D',h') meets the same rule as ip' , and if ft' = g, then both ip' and 
L%l){D',h') meet (i). 

(b) . This part follows from the definition of z. 

(c) . Set D = D' AG, which is a valid set of pairs. If ip' meets (i) or (iii), 
then the pair that is added to D' belongs to F, so j < g, and it follows from 
Lemma [4. 291 that l4>{D, ft') meets the same rule as ip' . Otherwise, the substitution 
rule decreases the level of -0', so ft' > fto + 1 > z. If ft' = g, then D' ~ D^, and 
it follows from Lemma [4.301 that the level of iij}{D,h') = tip{Dz,9) is decreased. 
We may therefore assume that ft' < g. If (ft', ft') ^ D', then Lemmas 14.291 and 
14.301 imply that the level of Lip{D, ft') is decreased, so assume that (ft', ft') G D'. 
Lemma [4.311 implies that L^{D,h') does not satisfy condition X. 

Write L^p{D,h') — {D,JI, S,h'). If the level of Lip{D,h') is not decreased, then 
this 4-tuple meets (iii) or (iv), and Lemma [4.291 implies that a pair from column g 
is added to D. It follows that D = D^U F and iip{D, ft') satisfies W{h',g). Notice 
that (z,g) G d^C, since otherwise we obtain z — h' — ho + 1, D = D, and fj, = fi, 
hence ip satisfies W{h',g), and this contradicts that the most recent predecessor 
of ip of level ft' does not meet (iii) or (iv). The definition of z therefore implies 
that W{z,g) fails for Lip{Dz,g), and the same is true for Lijj{D, ft'). Since the latter 
4-tuple satisfies W(ft',g), we deduce that JI^ — flj^, < h' — z. Now notice that 
Xz < JIz'i if z = TO then this follows because h' ~ m < g and hence (to — 1, to) G C. 
Lemma [4. 281 therefore implies that Xz — Xh'-i < Jlz—f^h' < h' — 'z, which contradicts 
the fact that A is /c-strict. 
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(d) . We first show that (ft,o,^o) E DtU F. If this is false, then we must have 
ho = g = TO, so the pair (m^m) was added to D by rule (i). This implies that 
jj^m > k and therefore Jim > k. Since Jli>Xi>k + m — i for all i < m, it follows 
that Lip satisfies W(i, to) for 1 < z < to, hence z = m + 1 > /iq, a contradiction. 

Write Lip{Dt U F, ho) — {Dt U F,JI, S, ho) and assume that both W{ho,g) and X 
fail for this 4-tuple. Then W{ho,g) also fails for tip. li h < ho, then W{ho,g) fails 
for ■0 as well, so the pair {ho,g) was added to D by rule (iv). It follows that the 
predecessor {D, /i, S, ho) of tp meets (v), which is a contradiction. 

We therefore have h = ho and {h, g) ^ S C Dt U F . Since l4> satisfies X we 
have JIh > A?i_i, and Lemma 13.61 applied to t^p shows that / = g. It follows that 
M/i ^ ^h-i, since otherwise we must have /J^ < ^h-i — Jj-h and (/i, /) = (h^g) G S", 
contradicting that Lip satisfies X. Now Lemma 13.61 implies that — g, so 

tip{Dt U F, ho) satisfies X anyway and meets (iii) or (iv). 

(e) . lih' > h, then the substitution rule decreases the level of tp', and Lemma l4.31l 
implies that the same thing happens to i'ip[D,h'). Finally, if /i' = h, then both 
ip' = ip and Ltp{D' , h') = Ltp satisfy condition X and meet (v). □ 

This completes the proof of Claim 2, and of Theorem [TJ 



5. Theta Polynomials 

5.1. In this section we develop the theory of theta polynomials systematically; the 
exposition is influenced by that in Macdonald's text [M] III. 8]. We shall show that 
these polynomials share many common features with the Schur Q-functions. One 
exception is that when fc > 0, we do not have a natural Hopf algebra structure. 

Given any power series X]i>o ^i^^ variable t and an integer sequence a = 

(ai, q;2, . . . , ai), we write Cq — Cq^Cqj • • • Ca^ and set Rca — CRa for any raising 
operator R. We will always work with power series with constant term 1, so that 
Cq = 1 and Ci = for z < 0. The formal identities ^ imply that the equations 

(18) JJ(1 - Rij) Ca = det{Cai+j~i)i.j 

and 



TT -, , Ca = Pfaffian(CQ,,Q.)i<j 



where 



_ 1 - Rl2 _ _„ „ _ 

t + ^12 

are valid in the polynomial ring Z[ci, C2, . . .]. 

Let X = (xi, X2, . . .) be a list of commuting independent variables and let A = 
P^{x) be the ring of symmetric functions in x. Consider the generating functions 

oo oo oo oo 

E{x]t)=W{l+x^t)^^er{x)f and H{x]t)^W ^^hr{x)f 

for the elementary and complete symmetric functions and h^, respectively. Fix 
an integer k>Q,\eiy= (j/i, . . . , y^), and for each r define dr = 'dr{x ; y) by 



= y^^qr^i{x)ei{y). 



i>0 
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We let r^*^) be the subring of A ® . . . , yk]^^ generated by the i?r-: 

=Z[79l,^2,^?3,•••]• 
Set Q{t) — ^r>a^rt^' ii^ern have 

i=l 3 = 1 

and hence 

2fc 

e(t)e(-t) = E{y ; ; -0 = E , 

m=0 

where denotes {yf, . . . ,yl). It follows that 

if r is odd 



i-iy/'er/^iy') ifr 



IS even. 



(19) (-^y^^^j = 

i-\-j—r 

In particular, when r = 2m > 2A;, equation (fTO)) gives 

m 

(20) C = 2E(-l)''''^m+»^™-.. 

4=1 

Definition 5.1. Given any fc-strict partition A, we let A^ be the strict partition 
obtained by removing the first k columns of A, and let A^ be the partition of boxes 
contained in the first k columns of A. 

k 



We say that a partition A is k-odd if all its parts which are greater than 2k are odd. 
Proposition 5.2. (a) The for A k-strict form a Z-basis ofV'-'^K 
(b) The for A k-odd form a Q-basis ofT^^^ T^*^) ®z Q- 

Proof. It follows from (I^Ul) that for any partition A, either A is fc-strict, or -dx is a 
Z- linear combination of the such that fi is fc-strict and ^ >- A (dominance order). 
Furthermore, we have 

'&\{x;y) = 'Yqx-a(x)ea{y), 

a 

the sum over all compositions a with < a; < fc for all i. If A is fc-strict, we deduce 
that the homogeneous summand of 'dx of lowest x-degree is equal to qx^{x)ex'^{y)- 
Part (a) follows because the set of all products qx^{x)ex2{y), given by fc-strict 
partitions A, is linearly independent over Z. 

Equation (fT9)) implies that i?2m G Q[i^i, ■ ■ ■ ,'&2m-i] for m > k. By induction 
on m it follows that '32m G Q[??i, • • • , t?2fe, ^92fe+i; ^^2fe+3, ■ ■ ■ ,'&2m-i] for all m > k, 

(k) 

hence the monomials dx indexed by fc-odd partitions A span Tq as a vector space 
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over Q. Finally, for each £ N, the number of k-odd partitions of d is equal to the 
number of /c-strict partitions of d, as verified by the equality of generating functions 

'2 k 

A fc-odd r=l r>k r r>k 

r— 1 r>k X fc-strict 

This completes the proof of the proposition. □ 
5.2. Recall the notation of i il.2l 

Definition 5.3. For any valid set of pairs D C A° and integer sequence A, define 
the polynomial &{D, A) by the raising operator formula &{D, A) = R^iix. Equiva- 
lently, we recursively set 

a 

where A = has length i and the sum is over all (_D, ^)-compatible vectors 

a G N^~^. For any fc-strict partition A, the theta polynomial Qx{x ;y) is defined by 
Ga = e(C(A),A) =i?^^A. 

Definition 15.31 and equation (|20p imply that each polynomial Qx can be written 
in the form 

where the sum is over fc-strict partitions /i >- A and G Z. We deduce from 
Proposition l5.2f a) that the polynomials 0a indexed by fc-strict partitions A form a 
Z-basis of rC^). 
Let 

H(IGfe) =liniH*(IG(n-fc,2n),Z) 

be the stable cohomology ring of IG; that is, the inverse limit in the category of 
graded rings of the system 

••• ^ H*(IG(n - fc,2n),Z) ^ H*(IG(n-|- 1 -k,2n + 2),Z) i 

From the presentation of H*(IG(n — fc, 2n), Z) given in jBKTll Thni. 1.2], we deduce 
that IHI(IGfc) is isomorphic to the polynomial ring Z[(Ti, tT2, ■ • •] modulo the relations 

m 

+ 2 y^(~l) Vm+tCTm-i — 
1=1 

for all m > k. Since the generators i^r of F^'^' satisfy (I^Ul) . we have a surjective ring 
homomorphism (f> : H(IGfc) — F^'^' sending to 19^ for each r. Theorem [T] implies 
that 4>{crx) = Oa for any fc-strict partition A. Since the 0a form a basis of F'^'^^ we 
conclude that (j) is an isomorphism. This completes the proof of Theorem [S] 

Remark 5.4. For any fc-strict partition A and formal power series c — J2i>o'^i^^' 
define the polynomial 0a (c) = R^c\. The 0a (c) are Giambelli polynomials for 
both the classical and quantum cohomology of isotropic Grassmannians (Theorem 
[Hand |BKT2| ) and appear in more general Giambelli formulas which hold in the 
equivariant cohomology ring of isotropic partial flag varieties |T3j . 
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5.3. Consider the analogues of the polynomials when the er{y) are replaced by 
complete symmetric functions hr (y) . Define for each r a function -dr = "drix ; y) by 

"^r = y^^qr~i{x)hi{y) 

i 

and set B{t) = X!r>o^^^'^- then have Q{t)B{—t) ~ 1, or equivalently, 

n 

(21) Y.^-lYdr^n-r=Q, n>\. 

r=0 

As in fM', 1.2, (2.9')], the equations (PT|) imply that for any partition A, 

(22) det {^x^+J-^) = det (^xr+j-.) ■ 

Here A' is the partition conjugate to A, i.e., A'^ = | \h > *} for all i. 

If fc = 0, then -dr — = Qr for every r > 0. Let (V) denote the partition 
(1,1,..., 1) of length r. 

Proposition 5.5. Assume that fc > 1 and r G N. Then dr{x\y) = 8(ir)(x;y). 

Proof. Observe that C{V) — 0. It follows from this, the identity (HH]), and equation 
(P^ that 

Equation (l?T|l and the Whitney sum formula prove that the polynomials dr = 
map to the Chern classes of the dual of the tautological subbundle 5 — ?• IG 
under the isomorphism of ij5.2l A Fieri rule for the products dr ■ Oa was obtained 
by Fragacz and Ratajski |FR| (see also |T3[ Ex. 4]). 

Proposition 5.6. The -ffx for A k- strict form a Q-basis o/Fq . 

Proof It is clear from the equations dm that F^*^) = Z[i9i, i?2, t?3, . . .]. Since 
•ffxix-^y) — J2a>o 1>^-o{x)ha{y), we deduce that if A is fc-strict, the homogeneous 

summand of i^a of lowest cc-degree is equal to qxi{x)h\2{y). Moreover, the set of 
products q\i{x)hx2{y) for all fc-strict partitions A is linearly independent over Q. 
The result now follows by a dimension count. □ 

Example 5.7. When fc = 1, we have 

3 631 - 2 - 5 i?3i + 4 d2ii - ^9iiii. 

We deduce that the i)\ for A fc-strict do not form a Z-basis of F*^*^). Furthermore, 
the transition matrix between the Q-bases {i?a} and {6a} of Fq^'' is not triangular 
with respect to the dominance order. 

5.4. We next introduce an analogue of the Schur 5-functions in the ring F'^'^^. 

Definition 5.8. For any two finite integer sequences A, /i, define the function 
^ r^*^) by setting 



^\/\,i^'^y) = dGt(?9A,-p,+j-«(a;;2/))j,j. 
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Assume that A and /i are two partitions. Then, arguing as in 'M, 1.5], the skew 
function S*^^^^ (a; ; y) is zero unless > /i^ for each i. The functions S'a/p(2;) :— 

S^°/^ix;y) are well known (see [HI 111.8, Ex. 7] and jWl Sec. 2.7]). We also let 

s\'/^'{y) = det(eA._^^.+j_j(y))ij 

denote the (ordinary) skew Schur polynomial in the variables y. We have that 
sy/f^'{y) = unless < Xi — < k for each i. The functions S'a/^(x) (respec- 
tively, SA'//i'(y)) s-re known to be linear combinations of Schur Q- functions Qu{x) 
(respectively, Schur 5-polynomials s^i{y)) with positive integer coefficients. 

Proposition 5.9. For any partitions X, fi with fi C X, we have 
(23) s[^/^{x;y) = ^ 5A/^(a;)s^,/^, (y) = ^ 5^/^(a;)sA'/i.' (y) 

summed over all partitions v such that fi C i' d X. 

Proof. Let x — (xi,a;2, . . .) be another infinite list of variables and define the ring 
A = Z[ei(x), e2(x), . . .] (K> Z[ei(?/), . . . , efc(?/)]. According to [M, 1.(5.10)], we have 

SA'/M'(^>y) ^^sy i^,i^)s^, |^,,{y) = ^s^, |^,,i^)sy i^,{y) 

in A. This is mapped to (|23p under the ring homomorphism A — >■ F^*^^ defined by 
sending 6^(5;) to qi{x) and ej(j/) to ej(y). □ 

The definition of S^^^ implies that S^y^^ = i^^a + X^^^a '^a^i^^ for some integers 
dAp, and therefore that the set of S'^'^'' for A fc-strict forms another Z-basis of r'^'"'^ 
For any integer sequence a and raising operator i?, set RSa^ — S^^^. The next 
result follows from the identity S^\x; y) — 9(0, A), which is derived from ([TSl) . 
Proposition 5.10. For any k-strict partition X, we have 

0a(x;2/)= n {^-R^l+Rl-■■■)Si'\x■,y). 

(«j)eC(A) 

5.5. In this section, we give the proof of Theorem [3l Let A be a fc-strict partition 
of length i. Note that if A.; + Xj < 2k + j — i for all i < j, then C(A) = 0, and we 

deduce from (fT8)) that 0a = s\^'' . Part (a) of the theorem then follows by setting 
/Lt = in ([23]) . 

Suppose now that A,; + Xj > 2k + j — i for all i < j < L Then A is a strict 
partition. For any strict partition /i C A with ^(/i) > £ — 1, we define the shifted 
skew shape 

S{X/^i) = {X + e,)/{^i + et), 

where u = (0, 1, 1). 

Following [HHl Chp. 9] and |Ml ULS, (8.8) and Ex. 8(c)], for any integer sequence 
7 of length the (generalized) Schur Q-function is defined by 

n TTf^'^- = ^V 

l<i<j<l 

Given any raising operator R, we have 

R-dx{x;y) = ■dR\{x;y) =^ea{y) qR\-a{x) = ^ea{y) Rq\-a(x). 

a a 
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It follows that 

(24) ©A = 51 ^"(2^) -R^^A-aCa;) - J2 QA-a(a;)e„(y). 

a a 

where the sums run over all compositions a with < < A; for each i. 

Since A is strict and Xe-i + > 2A; + 1, we see that A^ > Oi for all compositions 
a indexing the sum (p4)) and every i except possibly i — £. We deduce from [Mj 
III. 8, Ex. 8(c)] or Lemma [131 that is skew symmetric in 7 for all integer vectors 
7 = A — a which appear. It follows that we may rewrite ()24p as 

0A = 5] ^(-l)'^Q^(a;)eA-^(^)(y) 

summed over strict partitions with £(/Lt) e {£ — !,£}. Part (b) follows from this 
because 

(-l)"'eA_t„(^)(y) = det(eA,-p,(2/))i<,j<^ = s5(a/m)'(2/) • 

6. Schubert Polynomials for Isotropic Grassmannians 

6.1. The polynomials Q\{x;y) fall within the Billey-Haiman theory of type C 
Schubert polynomials €,w{x,z). We will prove and discuss this in detail in this 
section. Let Wn be the hyperoctahedral group of signed permutations on the set 
{1, . . . , n}, and define Woo = Un The group Woo is generated by the simple 
transpositions Si — + 1) for i > 1, and the sign change sq defined by sq{1) = 1 
and so{p) = p for p > 1. Let w 6 Woo- A reduced factorization of w is a product 
w — uv in Wn such that i{w) = l{u) + £{v). We say that w has a descent at 
position i if £{wsi) < £{w)] this is equivalent to the inequality w{i) > w{i + 1) if we 
set ^(O) — 0. The signed permutation w is called k- Grassmannian if k is the only 
descent position for w. 

The elements of Wn index the Schubert classes in the cohomology ring of the 
flag variety Sp2„ /-B, which contains H*(IG(n — fc, 2n),Z) as the subring spanned 
by Schubert classes given by fc- Grassmannian elements. In particular, each fc-strict 
partition A in V{k, n) corresponds to a fc- Grassmannian element w\ e Wn which 
we proceed to describe; more details and relations to other indexing conventions 
can be found in [Til §4] • 

Notice that a fc-strict partition A belongs to 'P(fc, n) if and only if its Young 
diagram fits inside the shape 11 obtained by attaching an (71 — fc) x fc rectangle to 
the left side of a staircase partition with n rows. When n — 7 and fc = 3, this shape 
looks as follows. 



n = 



The boxes of the staircase partition that are outside A are organized into south-west 
to north-east diagonals. Such a diagonal is called related if it is fc-related to one 
of the bottom boxes in the first fc columns of A, or to any box [1, i + 1] for which 



36 



ANDERS SKOVSTED BUCH, ANDREW KRESCH, AND HARRY TAMVAKIS 



Xi < i < k] the remaining diagonals are non-related. The /c-Grassmannian element 
for A is defined by 

wx = (ri,...,rfc,(Ai)i,...,(Ai)p,ui,...,'u„_fc_p), 

where ri < • • • < are the lengths of the related diagonals, p — £{X^) — ik{X), and 
Ml < • • • < Un-k-p arc the lengths of the non-related diagonals. For example, the 
partition A = (7, 4, 2) G 7^(3, 7) corresponds to the element wx = (2, 5, 6, 4, 1, 3, 7). 



A = 
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The signed permutation wx G Woo depends on A and fc, but is independent of n. 
Furthermore, if Ai < fc, then wx G is the type A Grassmannian permutation 
for the conjugate partition A' with descent at position k. 

6.2. A sequence a — (ai, . . . , a„i) is called unimodal if for some index r we have 

Q-l > 0-2 > ■ • • > Ir-l > flr < 0,r+l < ■ ■ ■ < Clm- 

A subsequence of a is any sequence (a^j , . . . , a^p) with 1 < ii < • ■ • < ip < to. 

Let w e Woo and let A be a strict partition such that |A| = £{w). A Kraskiewicz 
tableau |Kr| for w of shape A is a filling T of the boxes of A with nonnegative integers 
such that, if Ti is the sequence of integers in row i from left to right, then (a) the 
row word ... Ti is a reduced word for w; and (b) for each i, Ti is a unimodal 
subsequence of maximum length in the word T£(^x) ■ ■ ■ Ti+iTi. 

For each w G Woo one has a type C Stanley symmetric function (x) , which is 
a positive linear combination of Schur Q-functions }BH|, IFK| IL] . There exist several 
combinatorial interpretations for the coefficients in this expression. We will use a 
result of Lam [L] stating that 

(25) FUx)=Y.eiQxix) 

A 

where equals the number of Kraskiewicz tableaux for w of shape A. 

Example 6.1. Assume that fc = and let wx be the 0- Grassmannian element 
defined by a strict partition A. In this case there exists a unique Kraskiewicz 
tableau Tx for wx- This tableau has shape A and its ith row contains the integers 
between and A^ — 1 in decreasing order. For example, we have 



5 


4 


3 


2 


1 


o| 


4 


3 


2 


1 







1 














To see this, one checks that every reduced word for wx can be obtained from the row 
word of Tx by using the commuting relations SiSj = SjSi for \i — j\ > 2; condition 
(b) above then implies that any Kraskiewicz tableau for wx has the same top row 
as Tx, and the remaining rows are determined by induction on ^(A). We deduce 
that F.uj^{x) = Qxix). 
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6.3. Following Billey and Haiman, each w € Woo defines a type C Schubert poly- 
nomial z). Here z — (zi, Z2, . . .) is another infinite set of variables and each 
€w is a polynomial in the ring A = Z[qi{x), <Z2(a;), . . . ; zi, 2:2, . . .]. The polynomials 
£u) for w G Woo form a Z-basis of A, and their algebra agrees with the Schubert 
calculus on symplectic flag varieties Sp2„ /B, when n is sufhciently large. According 
to |BH[ Thm. 3], for any w € Wn we have 

(26) 

) = II F„(x)6,(z), 

uv—w 

summed over all reduced factorizations w = uv in Wn for which v G Sn- Here &v{z) 
denotes the type A Schubert polynomial of Lascoux and Schiitzenberger [LS| . 

We next show that each theta polynomial agrees with the Billey-Haiman 
Schubert polynomial indexed by the fc-Grassmannian element G Woo cor- 

responding to A = (r). It is easy to see that has a unique reduced ex- 

pression, given by W(r) = Sk-r+iSk-r+2 ■ ■ ■ Sk when 1 < r < fc, and by = 
Sr-k-iSr-k-2 ■ ■ ■ siSqSi ■ ■ ■ Sk when r > k + 1. It follows that if W(r) = uv is 
any reduced factorization of W(^t) with v G 6*00, then v — for some inte- 
ger i with < ? < fc. The type A Schubert polynomial for is given by 
(3^j.j(z) = ei(zi, . . . , Zfe), and (|25p implies that the type C Stanley symmetric 
function for u — W(^r}W^i^ is Fu{x) = qr-i{x). We conclude from p6|) that 

(27) 'tw^^^{x,z) ^'^qr^i{xi,X2,- . ■)e^{zi,. . . ,Zk) ='dr{x;z), 

i=0 

as required. Since the Schubert polynomials multiply like the Schubert classes 
on symplectic flag varieties, Theorem [T] and (|27|) imply the following result. 

Proposition 6.2. The ring F*^'') of theta polynomials is a subring of the ring of 
Billey-Haiman Schubert polynomials of type C. For every k- strict partition A we 
have Qx{x;y) = <Lwx i^iV)- 

Notice that Proposition 16. 21 mav be used to get a different proof of Theorem [2l 
Proposition 16.21 and (|26p imply that for every /c-strict partition A we have 

(28) Qx{x;y)= ^ Fu{x)&,{y) , 

UV — Wx 

where the sum over all reduced factorizations Wx = uv in Wco with v G Soo- The 
right factor v in any such factorization must be a Grassmannian permutation with 
descent at position k. In fact, it is not hard to check that the right reduced factors of 
wx that belong to 6*00 are exactly the permutations Wi, given by partitions v CZ X^. 
Since the Schubert polynomial &w,Xy) is equal to the Schur polynomial Su'{y), we 
deduce from (|25|) that 

(29) eA(x;2/) = ^e^,Q^(x)s,.(j/), 

where the sum is over partitions /i and v such that is strict and C A^, and e^^^ 
is the number of Kraskiewicz tableaux for w\w~^ of shape /i. This completes the 
proof of Theorem U) 

Corollary 6.3. Let X be a k-strict partition. 
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(a) The homogeneous summand of Q\{x ; y) of highest x-degree is the type C Stanley 
symmetric function Fw^{x), and satisfies Fw^{x) = R^q\{x). 

(b) The homogeneous summand ofO\{x;y) of lowest x-degree is Q\i{x) S(^x2y(jj). 

Proof. Part (a) is deduced by setting y = in (|28p and also in the raising operator 
expression eA(a; ; y) = R^'&x{x ; y). Part (b) follows from (HH), Exampleimi and the 
observation that w\w~^2 is the 0-Grassmannian Weyl group element corresponding 
to the strict partition A^. □ 

Example 6.4. Let k — 1 and A = (3, 2, 1), with corresponding Weyl group element 
wx = (4, 2,T, 3) e W4. Then we have 

©321 = (Q42 + Qz2l) + {Qil + 2 Q32) Sy + 2 Qai Siy + Q21 Sm' 

(with the variables x and y omitted). The terms in this expansion are accounted 
for by the Kraskiewicz tableaux in the following table. 



V 




Kraskiewicz tableaux for w\w^ ^ 





(4,2,1,3) 


3 
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0| 1 1 
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(2,4,T,3) 


3 1|0|2 
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3 T| 


|_0_ 
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(1,1) 


(2,T,4,3) 
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ilo 
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0|3 






0_ 




0_ 






(1,1,1) 


(2,1,3,4) 




1 








Remark 6.5. The polynomials 2~^''''^^@\ given by A:-strict partitions A multiply 
like the Schubert classes on odd orthogonal Grassmannians OG(n — k,2n + 1) 
and agree with the Billey-Haiman Schubert polynomials of type B indexed by k- 
Grassmannian elements w\. For the even orthogonal Grassmannians OG(n — A:, 2n), 
both the Giambelli formula and the corresponding family of polynomials are more 
involved; we plan to develop this theory elsewhere. 
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